Investigating L2 learners’ text-to-video resemiotisation in AI-enhanced digital multimodal composing

Digital multimodal composing (DMC) has garnered attention in second language (L2) writing classrooms. The introduction of artificial intelligence (AI) has been a game-changer in this field, providing tools that amplify DMC by translating text into images and videos. However, there is a research gap on how these tools are utilised by learners. This study aims to fill this gap by applying a resemiotisation perspective to examine how learners employ AI tools to integrate linguistic, semiotic, and technological elements in translating written genres into video formats. Conducted at a comprehensive university in China, this research involved 75 undergraduates in an English writing course. During the course, students used an AI-powered text-to-video platform called Pictory to convert technical proposals into videos. Pictory facilitated the students' video creation by enabling them to craft search queries to generate video clips. Data sources included students' technical proposal outlines, video composition plans, reflections on the text-to-video generation process, and the produced video compositions. Through a content analysis of students' reasons for revising search queries during their text-to-video conversion processes as well as a comparative analysis of their original and revised search texts alongside the multimodal elements within the resulting videos, the study revealed that text-to-video resemiotisation in AI-enhanced DMC involved students modifying written texts (transformation) and the AI technology converting them into videos (transduction), diverging from traditional DMC where both the transformation and transduction stages are overseen by human agents. Specifically, during the text-to-video resemiotisation processes, students implemented various customisation initiatives targeting search credibility, scope, relevance, and modality to generate suitable video clips. This study enriches our understanding of DMC in AI-enhanced learning contexts, providing insights for future DMC curriculum development that effectively leverages AI tools to improve learners' DMC skills.