Integrating a Scaffolding-Based, LLM-Driven Chatbot into Programming Education: A University Case Study

Enhancing programming education with Artificial Intelligence (AI) requires aligning technological tools with sound pedagogical principles. This study examines the effectiveness of a scaffolding-based chatbot, powered by Large Language Models (LLMs), that delivers multiple-choice questions (MCQs) to support student learning in an introductory Python programming course. Two groups of 40 undergraduate students participated: one interacted with a conventional chatbot, while the other used a modified version incorporating scaffolding strategies such as guided questioning and hinting. Pre- and post-tests were administered to assess students' conceptual understanding of Python programming, alongside a survey capturing students' perceptions of chatbot use in programming education. Results indicate that although both groups demonstrated improvement in their conceptual understanding, the scaffolding-based chatbot achieved significantly greater learning gains. Specifically, the post-test mean score for the scaffolding group was 26.55 ($\text{SD}=5.33$), compared to $20.65(\text{SD}=5.87)$ in the conventional group $(t(78)$ $=4.71 p<.001$). Compared to the conventional group, a higher percentage of students in the scaffolding group reported perceived benefits of the chatbot: improved conceptual understanding (70 % vs. 85%), enhanced problem-solving skills (30 % vs. 65 %), increased learning confidence (48 % vs. 78 %), and a lower percentage reporting concerns related to inaccuracies (68 % vs. 48 %) and overreliance (63 % vs. 43 %). These findings highlight the potential of integrating pedagogical scaffolding into chatbots to support more effective, studentcentred programming instruction.