Exploring LLM-Powered Interaction Modalities in Virtual Museums: Eye-Tracking and User Experience Insights

The integration of VR and large language models (LLMs) offers significant potential to enhance educational and cultural experiences. This study investigates how ChatGPT-driven interaction modes affect user experience and visual attention in a VR museum. Thirty-one participants engaged with three conditions: a Text and Audio Guide (TAG), a Limited Interactive Avatar (LIA), and a Fully Interactive Avatar (FIA) with animations and personalized dialogue. Using a within-subjects VR experiment, we collected subjective feedback and eye-tracking data. Results showed that FIA significantly increased user satisfaction, engagement, immersion, and visual focus compared to TAG and LIA, demonstrating the benefits of responsive, embodied AI. Furthermore, visual attention patterns strongly correlated with self-reported experience quality, suggesting attention is a reliable indicator of engagement. These findings provide practical design guidance for creating natural, context-aware, and emotionally expressive LLM-driven VR environments, advancing the development of more effective and engaging immersive learning systems.