Generative AI responses are a dime a dozen; Making them count is the challenge – Evaluating information presentation styles in healthcare chatbots using hierarchical Bayesian regression models

The emergence of large language models offers new opportunities to deliver effective healthcare information through web-based healthcare chatbots. Health information is often complex and technical, making it crucial to design human-AI interactions that effectively meet user needs. Employing a 2x2 between subjects design, we controlled for two independent variables: communication style (conversational vs. informative) and language style (technical vs. non-technical). We used hierarchical Bayesian regression models to assess the impact varying information presentation styles on effectiveness, trustworthiness, and usability. The findings revealed perceptions of low usability significantly decreased the effectiveness of the healthcare chatbot. Additionally, participants exposed to the conversational style of the chatbot had significantly increased likelihoods to perceive it with higher usability but were also more likely to be less trusting of the chatbot. These results indicate varying information presentation styles can impact user experience and offers insights for future research with healthcare chatbots and other AI systems.