The Effects of Cognitive Engagement and Humanness of Chatbots on Learning Outcomes and Motivation

According to the ICAP framework, engaging in constructive cognitive modes yields better learning outcomes than active modes. Also, prior studies on educational chatbots suggest that enhancing chatbot humanness can improve learning. However, these two ideas have not been fully explored together, especially within the context of text-based, disembodied chatbots. This study investigates the impact of the cognitive engagement modes (constructive vs active) and chatbot humanness (humanized vs non-humanized) on learning outcomes and five dimensions of learning motivation. We conducted a two-by-two factorial user experiment with 55 chatbot users. Data was analyzed through a mixed-method approach to examine the main and interaction effects of the two independent variables. Regarding learning outcomes, our data showed that learners who interacted with constructive chatbots showed higher learning outcomes than those who interacted with active chatbots. In addition, learners who interacted with non-humanized chatbots reported higher learning outcomes than those who interacted with humanized chatbots. Lastly, we observed a significant interaction effect between the two independent variables on tension-pressure and perceived competence, which are two dimensions of learning motivation. Our study extended the applicability of the ICAP framework to the domain of chatbot-based learning, challenged the assumption that the humanness of chatbots can lead to improved learning outcomes, and underscored the importance of exploring both the cognitive engagement modes and the humanness of chatbots when designing chatbots to enhance users' learning motivation.