Metrics of Success: Evaluating User Satisfaction in AI Chatbots

The rapid advancement of Artificial Intelligence (AI), particularly through Large Language Models (LLMs), has catalysed a technological revolution, leading to the widespread adoption of AI-driven chatbots across industries. OpenAI’s customisable generative pre-trained transformer (GPT) offerings have popularised generative AI, enabling organisations of all sizes to implement chatbots for customer support. This development presents an opportunity for businesses to offer 24/7, cost-efficient customer service that can overcome the historical limitations of chatbots that lack a "human element." However, despite the proliferation of AI chatbots, there remains a crucial need to evaluate their effectiveness in meeting user needs and preferences for human-like interaction. Current service quality assessment tools, such as SERVQUAL and E-SERVQUAL, are unable to evaluate AI-specific capabilities like language intelligence and recognition. Existing research also lacks information on factors that affect user satisfaction and the continued use of AI chatbots. Based on a mixed-methods study, this paper proposes a new instrument for measuring user satisfaction with AI chatbots, specifically for customer support roles. Using the Stanford five-step Design Thinking Process, this study devised a customer support AI chatbot evaluation instrument through a literature review, Cheatstorming, and SCAMPER techniques, followed by testing in a Danish company. The research employs Prentice and Nguyen’s three-stage scale development process to ensure content, reliability, and construct validity, addressing gaps in current scholarship and advancing understanding of AI chatbot user satisfaction.