Transforming Learning or Empty Promise? A Meta-Analysis of Generative AI in Education

Abstract This meta-analysis examines the impact of generative artificial intelligence (GenAI) tools, such as ChatGPT, on students’ academic achievement. Drawing on 52 experimental and quasi-experimental studies across educational levels and domains, we synthesized evidence from interventions using GenAI to support learning. Eligible studies reported performance outcomes (e.g., test scores, grades, GPA) and met rigorous inclusion criteria. Overall, GenAI-based instruction showed a positive effect (Hedges' g = 1.193) on academic achievement, with substantial between-study variability indicating that GenAI’s effectiveness depends on contextual and design features. Moderator analyses identified two significant factors, which are instructional role and subject area. GenAI was most effective when used to support formative functions such as assessment, feedback, and tutoring, suggesting that its strength lies in providing adaptive guidance and personalized learning support. Effects also varied across subject areas. Language education showed the strongest and most consistent gains, reflecting a close alignment between GenAI’s natural language capabilities and core instructional practices. In contrast, more modest effects were observed in computer science and art education, where applications tend to be narrower in scope. Other moderators, including educational level, sample size, intervention duration, and learning domain, did not yield statistically significant differences but revealed descriptive patterns that may inform future research and implementation. These findings suggest that GenAI tools hold considerable promise for improving academic performance when thoughtfully integrated into instructional practice. Educators and policymakers should consider both the role GenAI plays and the subject context to ensure its effective use in diverse educational settings.