Influence on Judgements of Learning Given Perceived AI Annotations

In this study, we designed a tool to investigate the relationship between students' ability to render accurate judgements of learning (JOLs) with decision-making behavior when annotating their own work and comparing it with perceived AI-generated annotations. Our findings suggest that students rarely adjust their JOLs after seeing the AI annotations, indicative of a strong self-confirmation bias. Trust in the AI tool was associated with a decreased likelihood of changing initial judgments, in part due to the similarity of AI annotations with their own. The process of using the tool to self-annotate was found to enhance performance on a post-test. Emphasizing clear learning objectives and being transparent with the limitations of AI functions may improve the effectiveness of such tools as a way to provide quick feedback and mitigate hesitancy towards wider-scale adoption.