L2 writers’ critical AI literacy in AI-assisted academic writing: scale development and validation

Abstract The proliferation of generative AI (GenAI) in academic writing necessitates that second language (L2) writers develop Critical AI Literacy (CAIL) – a set of competencies extending beyond operational skills to include ethical, strategic, and critical engagement. While conceptual frameworks for CAIL are emerging, a validated instrument to empirically measure this multifaceted construct for L2 writers has been conspicuously absent. This study addresses this gap by detailing the development and validation of the L2 Writers’ Critical AI Literacy Scale (L2-CAILS). Following established psychometric procedures, the scale was developed through literature review, expert validation, and pilot testing, and was subsequently administered to 420 L2 university students. The data were subjected to both exploratory (EFA) and confirmatory factor analysis (CFA) to establish construct validity and reliability. The analysis yielded a robust and reliable 25-item, five-factor model of CAIL: (1) Awareness and Understanding of AI, (2) Ethical and Responsible Use, (3) AI Application Strategies, (4) Critical Evaluation of AI Outputs, and (5) Metacognitive Engagement and Self-Regulation. The L2-CAILS provides a validated tool for researchers and educators to assess L2 writers’ competencies, design targeted pedagogical interventions, and conduct empirical research on the role of CAIL in AI-assisted academic writing.