Unraveling the Impact: An Empirical Investigation of ChatGPT’s Exclusion from Stack Overflow

This study explores the implications of a policy change announced by Stack Overflow, a popular question-and-answer platform, to prohibit the use of ChatGPT for generating questions or answers. To analyze data from Stack Overflow and Reddit, we employ the difference-in-differences (DID) empirical strategy to examine the impact of Stack Overflow’s policy. Our analysis focuses on the linguistic characteristics of content using natural language processing (NLP) techniques and voting features on both platforms. The results indicate that, after the restriction, the users on Stack Overflow exhibited a change in their answer generation, producing answers that exhibited more language complexity, positivity, and a larger number of words than the users on the AskProgramming subreddit. However, there was no discernible impact on the questions asked on Stack Overflow. Notably, the answers on Stack Overflow received more upvotes after the restriction, suggesting that the community perceived them to be of higher quality than answers on the subreddit. The frequency analysis reveals a noticeable decrease in the number of questions and answers posted by users on Stack Overflow following the restriction. Additionally, our supplementary analysis suggests a decrease in first-time contributors on Stack Overflow after the restriction. We validate our findings using an independent natural experiment leveraging Italy’s temporary nationwide ChatGPT ban, as well as a scenario-based experiment with 440 participants that provides direct evidence of compensatory knowledge signaling by contributors when AI-generated content is restricted. This study contributes to the existing literature by providing empirical evidence and insights into the decision to prohibit ChatGPT on a question-and-answer platform and provides practical implications to platform managers, informing them about the impact of the restriction on the change in answer content.