Enhancing Trust in LLM Chatbots for Workplace Support through User Experience Design and Prompt Engineering

Traditional chatbots have been essential for workplace support. With the rise in Large Language Model (LLM) chatbots and their quick and efficient solution to user queries, this new generation of chatbots will soon enter the field of workplace support to assist with IT, HR, and general workplace queries. However, trust concerns with LLM chatbots, which arise from factual errors, inaccuracies, and suboptimal response formatting, have become prominent and will be particularly critical in professional settings such as employee support within a company. This paper investigates factors influencing user trust in AI chatbots for workplace support, proposing solutions through UX design improvement and prompt engineering experiments. We conducted mixed-method user research to study the impact of response formatting and presentation on user trust and experience. Our qualitative user interviews and contextual inquiries aim to understand users’ expectations of these chatbots and their perspective of usage, followed by user surveys that validate users’ preferences through quantitative measures. The findings reveal that trust challenges arise from a perceived lack of credibility and transparency as a result of hallucinations, as well as concerns about data privacy. They also show the need for improved chatbot conversational experiences with more human-likeness, better contextual understanding abilities, and higher flexibility in input and output formats. To address these challenges, our research uniquely proposes and implements a solution based on the interception of UX design and prompt engineering. Actionable UX design implications for a trustworthy interface are outlined, along with prompt engineering solutions demonstrated through a prototype. This research contributes to the evolution of AI-driven chatbot technology, aligning with the broader goal of enhancing user satisfaction and trust in automated support systems. This paper provides valuable insights for AI chatbot developers, designers, and researchers to meet the critical need for effective and reliable chatbots tailored to workplace support. This study also points to opportunities for future research topics around trustworthiness in Artificial Intelligence to explore how diversity, technology, research design, and ethical aspects would factor into user trust and experience.