How Generative AI Transforms Questioning Behavior on Q&A Platforms: Evidence from a Natural Experiment with Pilot Usage of ChatGPT

The rapid advancement of generative AI has enabled the automated generation of answers, introducing new dynamics in user behavior related to question-asking on digital platforms. However, the impact of users’ experience with generative AI for answer generation on their questioning behavior remains unclear. On the one hand, the availability of generative AI tools may lead users to rely on these tools for formulating questions, potentially resulting in fewer but higher-quality inquiries. On the other hand, using generative AI may motivate users to ask more questions to enhance their knowledge and build social reputation within the community. We investigate how the use of generative AI, specifically ChatGPT, to generate answers affects users’ questioning behavior on Stack Overflow, one of the largest Q&A platforms. Leveraging a natural experiment involving the introduction and subsequent ban of ChatGPT on the platform, we apply a difference-in-differences (DID) estimation approach to assess the effects of generative AI usage on the quantity and quality of user-generated questions. We measure these outcomes through metrics such as the number of questions posted per week, question length, novelty, and the number of received upvotes. Our findings reveal that the use of generative AI to generate answers is associated with an increase in the number of questions, which feature longer, more novel content and receive more upvotes. We examine two mechanisms: complexity shift, where ChatGPT handles routine queries while more advanced queries remain for the community, and skill enhancement, where users learn from ChatGPT to ask clearer questions. We also test two moderators: answer volume (proxy for user experience) and platform reputation score (proxy for user reputation). Results show weaker effects among experienced answerers but stronger effects among high reputation users. These insights highlight how generative AI shapes engagement and can guide platform strategies for AI integration.