Large language models and rheumatology: a comparative evaluation

In recent months various large language models (LLMs) have shown substantial potential across diverse fields, including medicine. 1 Venerito V Bilgin E Iannone F Kiraz S AI am a rheumatologist: a practical primer to large language models for rheumatologists. Rheumatology (Oxford). 2023; (published online June 12.)https://doi.org/10.1093/rheumatology/kead291 Google Scholar GPT-4 by OpenAI is the most well-known LLM. Yet, alternatives exist, such as AnthropicAI's Claude and Google Bard, an online assistant powered by Google's Pathways Language Model (PaLM 2). 2 Chowdhery A Narang S Devlin J et al. PaLM: scaling language modeling with pathways. arXiv. 2022; (published online Oct 5.) (preprint).https://doi.org/10.48550/arXiv.2204.02311 Google Scholar However, a gap remains in comparative studies examining the performance of these diverse LLMs in rheumatology contexts. Our study seeks to fill this gap by evaluating the performance of three widely used LLMs: OpenAI's GPT-4, AnthropicAI's Claude, and Google's Bard. Notably, Bard possesses the unique capability of accessing real-time internet data, theoretically enabling it to generate more diverse and up-to-date responses. Access to OpenAI's GPT-4 and Google Bard was obtained via the web, whereas AnthropicAI Claude version 1.3 was accessed via slack between June 23 and 24, 2023. Given the release of Claude version 2 on July 11, 2023, the iteration was repeated on July 15, 2023, via the web.