Artificial intelligence literacy in assessment: Empowering pre‐service teachers to design effective exam questions for language learning

Abstract The role of artificial intelligence (AI) in education plays a crucial role in teacher training digitalisation. Although AI has enormous potential, not much is known about how pre‐service teachers perceive and utilise AI tools in professional practice. Hence, this study, guided by the Unified Theory of Acceptance and Use of Technology framework, investigates pre‐service English as a foreign language teachers’ experiences using MagicSchool, an AI‐based educational tool, to design exam questions, aiming to explore how AI tools can enhance assessment practices in teacher education. Participants were 27 fourth‐year pre‐service teachers. Data for this case study were collected through semi‐structured interviews and reflective reports and subsequently subjected to thematic analysis. The findings reveal that MagicSchool improved time efficiency and reinforced the creation of various question types. Participants also mentioned its practicality in generating rubrics and materials for varied proficiency levels. However, challenges such as crafting effective prompts, verifying content and addressing cultural or contextual mismatches were recognised. Moreover, ethical concerns, such as plagiarism and minimised creativity, were highlighted, with participants warning against over‐reliance on AI. The study underscores the potential of AI in exam preparation while emphasising challenges, advocating for a balanced approach that integrates AI responsibly. Implications for teacher education include fostering AI literacy, promoting critical engagement with AI‐generated content and ensuring ethical and pedagogically sound implementation in assessment design.