Chinese intermediate English learners outdid ChatGPT in deep cohesion: Evidence from English narrative writing

ChatGPT is a publicly available chatbot that can quickly generate texts on given topics, but it remains unknown whether the chatbot is as good as or even superior to human writers in all aspects of writing, and whether the chatbot's writing quality can be significantly improved by updating commands. To investigate these issues, this study compared writing performance on a narrative topic by ChatGPT and Chinese intermediate English (CIE) learners so as to reveal the chatbot's advantages and disadvantages in writing. The data were analyzed by virtue of the five discourse components using Coh-Metrix (a special instrument for analyzing language discourse), and the results revealed that ChatGPT performed better than human writers in narrativity, word concreteness, and referential cohesion, but worse in syntactic simplicity and deep cohesion in its initial version. After its commands were updated, the resulting version was better in its syntactic simplicity, but it still lagged far behind the CIE learners' writing in deep cohesion. In addition, a correlation analysis of the discourse components suggests that narrativity was correlated with referential cohesion for both ChatGPT and the human writers, but the correlations varied within each group.