Comparative evaluation of artificial intelligence language models on knowledge of dental antibiotic use

BACKGROUND: Artificial intelligence–based large language models (LLMs) are increasingly being used across multiple disciplines. In healthcare, as well as in education, research, and information access, the quality, reliability, and accuracy of the responses generated by these models have become a growing concern. This study aimed to comparatively evaluate the ability of two prominent LLMs, ChatGPT-3.5 and Gemini-1.0, using the most advanced publicly accessible versions at the time of AI access (February 9, 2025), to inform both the public and dental professionals about antibiotic use in dentistry, and to assess the content quality, accuracy, and comprehensiveness of their responses. METHODS: A total of 36 questions—comprising 12 multiple-choice, 12 true/false, and 12 open-ended items—were developed and posed to both ChatGPT-3.5 and Gemini-1.0, which were the most advanced publicly accessible versions at the time of AI access (February 9, 2025). The responses were independently scored by four expert endodontist dentists on a scale of 1 to 5. Data analyses were performed using SPSS. In addition to descriptive statistics, the Wilcoxon signed-rank test was used to compare model scores for identical questions, the Kruskal–Wallis test was applied to examine score differences across question types, and the intraclass correlation coefficient (ICC) was calculated to assess inter-rater reliability. RESULTS: Overall, responses generated by ChatGPT-3.5 and Gemini-1.0 were evaluated, and Gemini’s responses received higher scores than those of ChatGPT. The mean score for ChatGPT was 4.08 ± 0.83 (median = 4.0, interquartile range [IQR] = 1.25), whereas the mean score for Gemini was 4.65 ± 0.58 (median = 5.0, IQR = 1.00), indicating an approximate 0.57-point difference in favor of Gemini. The difference between the two models was statistically significant based on the Wilcoxon signed-rank test (p < 0.05). When analyzed by question type, Gemini scored higher than ChatGPT across all categories, including multiple-choice, true/false, and open-ended questions. CONCLUSIONS: While large language models (LLMs) and AI chatbots, specifically ChatGPT-3.5 and Gemini-1.0, demonstrate reasonable performance in providing basic knowledge about antibiotic use in dentistry, their reliability remains limited in areas directly influencing clinical decision-making—particularly those involving patient-specific contraindications and individualized treatment considerations. These findings should be interpreted in the context of the specific model versions evaluated in this study. As LLMs continue to evolve rapidly, future studies using updated models and real-world clinical validation are warranted to further clarify their role as decision-support tools rather than standalone authorities.