Human development integrates various cognitive and emotional functions, among them empathy, which involves compassion, kindness, and caring for others. The development of empathy emerges in the initial moments of dyadic interactions with carers (mostly with a mother), parents, relatives, and eventually with primary school teachers. Currently, AI technology in classrooms may destabilise teacher-student communication and the development of empathy. To compensate for this potential loss, computational researchers are extensively training AI technologies like chatbots to show empathy in human-machine interactions. This creates an ‘artificial empathy’ based on human-templated language. Thus, this article examines the language use in a conversation between a human and a chatbot, like ChatGPT, in reference to empathetic markers as part of a case study. The theoretical background is inspired by affective neuroscience to frame empathy, Edda Weigand’s Mixed-Game Model to explore dialogue, and the concept of embodiment that incorporates mind-body dynamics. This study suggests that ‘artificial empathy’ should be a more precise description of human-chatbot interactions to avoid anthropomorphising generative AI and to retain teachers as ethos and role models for the growth of young pupils’ moral and ethical learning.