Human perceptions of register alignment and functional appropriateness: ChatGPT-generated vs human-authored texts

Abstract This study investigates how human participants evaluate the human likeness of Chat Generative Pretrained Transformer (ChatGPT)-generated texts compared to human-authored texts in terms of register alignment and functional appropriateness. It involved eighty-three participants who evaluated four texts (two by ChatGPT and two by humans) representing university textbook and blog registers. Participants answered questions about text authorship, register identification and alignment, and purpose identification and functional appropriateness. Findings showed that participants struggled with identifying ChatGPT-generated texts accurately, with only one ChatGPT text correctly identified by the majority, while they performed better in identifying human-authored texts. Participants were also challenged in identifying the correct register, with no notable differences between human-authored and ChatGPT-generated texts. However, participants generally rated all texts as aligning with their chosen registers, with texts identified as human-authored perceived as more aligned. Moreover, although participants had difficulty distinguishing between human and artificial intelligence (AI)-generated texts and identifying registers, they could successfully recognize the communicative purposes of texts regardless of authorship, which suggests that ChatGPT can convey the communicative purpose of a text. Nonetheless, texts identified as human-authored were perceived as more functionally appropriate. This study provides insights into the current state of AI-generated text quality. The findings suggest caution when considering the integration of ChatGPT-generated texts into language education. While ChatGPT shows potential in providing language practice and development, the perceived lower quality in terms of register alignment and functional appropriateness may affect its effectiveness. Educators should critically evaluate the use of AI-generated texts to ensure they meet the linguistic standards required for educational purposes.