Capabilities of Large Language Models in Detecting and Managing Drug Interactions During Medication Reviews: Potential Implications as A Digital Assistant for Pragmatic Pharmacy Practice in Thailand

ABSTRACT Introduction The emerging capabilities of large language models (LLMs) have drawn increasing attention across various fields, including pharmacy practice in Thailand. Given the extensive number of available medications and the complex, often perplexing prescribing patterns, medication review remains a critical responsibility for pharmacists. This study explored the potential role of LLMs in supporting the medication review process within the Thai health care context, specifically focusing on their ability to detect drug interactions (DIs) and suggest context‐sensitive management strategies. Methods Ten clinical vignettes were constructed, each depicting a patient with a specific drug regimen seeking assistance from a pharmacist. These cases were tailored to reflect the Thai context and represent commonly encountered DIs in Thailand. Each vignette was submitted to a set of LLMs—ChatGPT‐4, ChatGPT‐4o, ChatGPT‐4o mini, Gemini 1.5, Claude 3.5, Microsoft Copilot, and Alisa 3.0—in both English and Thai. Evaluation metrics were developed and validated using the Index of Item‐Objective Congruence. Two independent evaluators assessed all responses, and inter‐rater reliability was measured using weighted Cohen's κ . LLM performance was scored based on percentage ranges, and cumulative scores were reported across evaluation domains. Results The weighted Cohen's κ values across six domains—(A) ability to identify DIs, (B) completeness, (C) clarity, (D) citation reliability, (E) usefulness, and (F) ability to assess harm—exceeded 0.6, indicating substantial inter‐rater agreement. All LLMs showed clinically acceptable performance in both languages. Citation reliability was limited in Alisa 3.0 and Gemini 1.5, while ChatGPT‐4o demonstrated the most consistent and well‐rounded performance. Conclusion The selected LLMs demonstrated their potential as capable digital assistants in medication reviews, although some models require further improvement and careful consideration when applied in real‐world settings. Nevertheless, human oversight remains essential; when used in parallel, LLMs and health professionals can work synergistically to enhance patient outcomes.