Development and Validation of an AI Literacy Diagnostic Tool for Middle School Students

Most artificial intelligence (AI) literacy diagnostic tools for middle school students are self-report questionnaires, making it difficult to use them as reliable assessments, while few such instruments provide informative feedback based on the assessment results. Therefore, in this study, we developed an AI literacy framework tailored for middle school students, which was validated through a Delphi survey of 15 experts. Based on this framework, we created a diagnostic tool consisting of 5 modules and 28 items and conducted a pilot test with 517 students. The results, analyzed using classical test theory and item response theory, indicated that all the items were appropriate and reliable. Nine experts participated in setting four proficiency levels – advanced, moderate, basic, and limited – using the Modified Angoff method. In addition, an experimental analysis was conducted using the pilot data to identify preliminary findings and explore how the AI literacy assessment tool could be applied in practice.