A Mixed-Method Study on the Effectiveness of Using ChatGPT in Academic Writing and Students’ Perceived Experiences

Introduction: The advent of artificial intelligence in education has brought forward tools like ChatGPT, which can potentially enhance students' academic writing abilities. However, there is limited empirical evidence examining its effectiveness and students' perceptions of its utility in academic contexts. Purpose: This study aimed to measure the effect of using ChatGPT on students' academic writing abilities and to investigate students’ perceived experiences regarding the use of ChatGPT in their writing process. Method: An explanatory mixed-method design was employed, incorporating a quantitative experiment followed by a qualitative investigation. The quantitative phase involved 102 fifth-semester students from an English education department at a university in Indonesia. These students were randomized into clusters based on their proximate writing test scores, resulting in two homogenous classes of 25 students each. These classes were then assigned to either an experimental group, which received 14 sessions using ChatGPT as a learning tool for academic writing, or a control group, which received 14 sessions using non-generative tools. Pre-tests and post-tests were administered to both groups. The qualitative phase involved interviews with 10 selected students from the experimental group to explore their perceived experiences with ChatGPT. Results: The pre-test scores indicated homogeneity between the experimental and control groups, with scores of 57.15 and 56.35 respectively. After the intervention, the post-test scores revealed significant improvement in the experimental group, with an average score of 81.11 compared to 60.30 in the control group. Statistical analysis demonstrated a significant disparity between the two groups (p-value = 0.0000 < 0.05 for the experimental group and p-value = 0.067 > 0.05 for the control group), suggesting that the use of ChatGPT significantly enhanced students' academic writing abilities. The qualitative findings supported these results, with students reporting that ChatGPT facilitated idea generation, organization, and construction in their writing process. Conclusion: The study concludes that ChatGPT significantly improves students' academic writing abilities, as evidenced by both quantitative and qualitative data. The tool's capacity to assist in the formulation and organization of ideas presents substantial potential for its use in academic research and writing. Given these findings, ChatGPT could be a valuable addition to the educational toolkit for enhancing academic writing skills.