AI or human? A study of university students’ awareness of library reference service agents

Purpose This study aims to examine university students’ awareness of artificial intelligence (AI) acting as the agent in university libraries’ reference services. It also aims to identify and discover factors that influence students’ judgement. Design/methodology/approach A within-participant design experiment was designed for this study. Five tasks covering five disciplines were developed. Two versions (ChatGPT versus subject librarian) of feedback were generated for each task. Participants were asked to judge the identity of the agent (a simplified Turing test) for each task. Think-aloud protocols were used to further analyse the factors affecting students’ judgements. Findings This study indicates students’ limited ability to distinguish AI and human agents. They are more likely to mistake librarians for AI agents with complicated tasks. The perceived knowledge, ability and comprehensibility of the AI system and the provided information quality and expression approach, together with AI usage experience, significantly affect students’ awareness of AI. Students associate objective, thorough and expert knowledge with AI, while detailed, vivid and colloquial explanations to librarians. Originality/value This study provides valuable insights into students’ awareness of AI versus human agents in reference services by synthesizing a theoretical model to explain students’ AI awareness, specifying components for each element and their relationships. It also benefits the effective integration of AI technology in libraries, especially in reference services, by emphasizing the importance of keeping pace with the development of AI and improving students’ AI literacy.