A Context-based Chatbot Surpasses Radiologists and Generic ChatGPT in Following the ACR Appropriateness Guidelines

Background Radiologic imaging guidelines are crucial for accurate diagnosis and optimal patient care as they result in standardized decisions and thus reduce inappropriate imaging studies. Purpose To investigate the potential to support clinical decision-making using an interactive chatbot designed to provide personalized imaging recommendations from American College of Radiology (ACR) appropriateness criteria documents using semantic similarity processing. Materials and Methods The authors used 209 ACR appropriateness criteria documents as a specialized knowledge base and used LlamaIndex, a framework for connecting large language models with external data, and ChatGPT-3.5-turbo to create an appropriateness criteria context aware chatbot (accGPT). Fifty clinical case files were used to compare the performance of accGPT with that of general radiologists at varying experience levels and to generic ChatGPT-3.5 and 4.0. Results The performance of all chatbots reached at least that of humans. For the 50 case files, accGPT performed best in providing correct recommendations that were "usually appropriate" according to the ACR criteria and also provided the highest proportion of consistently correct answers in comparison with the generic chatbots and radiologists. Furthermore, the chatbots provided substantial time and cost savings, with an average decision time of 5 minutes and a cost of €0.19 ($0.21) for all cases, compared with 50 minutes and €29.99 ($33.24) for radiologists (both P < .01). Conclusion ChatGPT-based algorithms have the potential to substantially improve the decision-making for clinical imaging studies in accordance with ACR guidelines. Specifically, the performance of a context-based algorithm was superior to that of its generic counterpart, demonstrating the value of tailoring artificial intelligence solutions to specific health care applications. © RSNA, 2023 Supplemental material is available for this article.