Enhancing Self-Regulated Learning Using Generative AI: Development and Evaluation of SRLMentor

Effective self-regulation is crucial for student success; however, many students struggle with it, especially during online activities, leading to disengagement. Existing methods to boost self-regulated learning (SRL) skills, such as writing self-reflective reports and using prompts in video lectures, lack timely, personalized feedback and can be labor-intensive. This study implemented the Self-Regulated Learning AI Mentor (SRLMentor), a system designed to support students' SRL skills, including goal-setting, planning, help seeking, and reflection. SRLMentor comprises three integrated modules to provide timely customized SRL feedback to each student. The first module features an SRL knowledge base that stores user chat records and relevant memory-driven adaptive SRL prompts, addressing a significant limitation of large language models—inability to store new experiences in long-term memory during a dialogue. The second module incorporates a Retrieval Augmented Generation (RAG) component to reduce content hallucinations, ensuring that students receive accurate information. The third module provides in-context learning examples that instruct the AI-based chatbot system to produce relevant SRL responses. We evaluated SRLMentor in an eight-session course with 25 students. Our assessment focused on RAG's performance in terms of factual consistency, answer correctness, and semantic similarity; accuracy of SRLMentor's detection of students' goals and plans compared to human coders; quality of SRLMentor's feedback; and students' perceptions of the system's usefulness. The results revealed that RAG enhanced factual, correctness, and semantic accuracy of responses. Additionally, SRLMentor's assessments of students' goals and plans closely matched those of human coders. The cluster analysis revealed that students who engaged more with SRLMentor exhibited greater improvement in SRL skills and course knowledge compared to those who engaged less frequently with the system.