Clinical Validation of Medical-based Large Language Model Chatbots on Ophthalmic Patient Queries with LLM-based Evaluation

作者
Ting Fang Tan,Kabilan Elangovan,Andreas Pollreisz,Kevin Bryan Dy,Wei Yan Ng,Joy Le Yi Wong,Liyuan Jin,Chrystie Quek Wan Ning,Ashley Shuen Ying Hong,Arun James Thirunavukarasu,Shelley Yin-His Chang,Jie Yao,D. C. Hong,Wang Zhaoran,Amrita Das Gupta,Daniel Shu Wei Ting
出处
期刊:Cornell University - arXiv [Cornell University]
被引量:1
标识
DOI:10.48550/arxiv.2602.05381
摘要

Domain specific large language models are increasingly used to support patient education, triage, and clinical decision making in ophthalmology, making rigorous evaluation essential to ensure safety and accuracy. This study evaluated four small medical LLMs Meerkat-7B, BioMistral-7B, OpenBioLLM-8B, and MedLLaMA3-v20 in answering ophthalmology related patient queries and assessed the feasibility of LLM based evaluation against clinician grading. In this cross sectional study, 180 ophthalmology patient queries were answered by each model, generating 2160 responses. Models were selected for parameter sizes under 10 billion to enable resource efficient deployment. Responses were evaluated by three ophthalmologists of differing seniority and by GPT-4-Turbo using the S.C.O.R.E. framework assessing safety, consensus and context, objectivity, reproducibility, and explainability, with ratings assigned on a five point Likert scale. Agreement between LLM and clinician grading was assessed using Spearman rank correlation, Kendall tau statistics, and kernel density estimate analyses. Meerkat-7B achieved the highest performance with mean scores of 3.44 from Senior Consultants, 4.08 from Consultants, and 4.18 from Residents. MedLLaMA3-v20 performed poorest, with 25.5 percent of responses containing hallucinations or clinically misleading content, including fabricated terminology. GPT-4-Turbo grading showed strong alignment with clinician assessments overall, with Spearman rho of 0.80 and Kendall tau of 0.67, though Senior Consultants graded more conservatively. Overall, medical LLMs demonstrated potential for safe ophthalmic question answering, but gaps remained in clinical depth and consensus, supporting the feasibility of LLM based evaluation for large scale benchmarking and the need for hybrid automated and clinician review frameworks to guide safe clinical deployment.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
Rony发布了新的文献求助10
1秒前
颜夜完成签到,获得积分10
1秒前
端庄向彤完成签到 ,获得积分10
1秒前
levitt233完成签到,获得积分10
1秒前
朴实的凡阳完成签到,获得积分10
2秒前
小科完成签到,获得积分10
2秒前
小何先生完成签到 ,获得积分10
2秒前
阳光的萃完成签到,获得积分10
2秒前
领导范儿的应助被mimosa采纳,获得10
2秒前
华仔的应助被打不死的小强采纳,获得10
2秒前
汉堡包的应助被小巴采纳,获得10
3秒前
JOE完成签到,获得积分10
4秒前
liuuuuuu完成签到,获得积分10
4秒前
黄明月完成签到 ,获得积分10
5秒前
JURIIY完成签到,获得积分10
5秒前
xiaofenzi完成签到 ,获得积分10
6秒前
绿兔子完成签到,获得积分10
6秒前
Lz发布了新的文献求助10
6秒前
科研狗的应助被独特的追命采纳,获得50
7秒前
想笑的老锅完成签到,获得积分10
7秒前
活力的秋灵完成签到,获得积分10
7秒前
huoguo完成签到,获得积分10
7秒前
1111111完成签到,获得积分10
7秒前
荣初露完成签到,获得积分10
7秒前
7秒前
8秒前
boxi完成签到,获得积分0
8秒前
TcsnAIj完成签到,获得积分10
8秒前
page完成签到,获得积分10
8秒前
小碗菌完成签到,获得积分10
8秒前
无情静柏完成签到 ,获得积分10
8秒前
菜鸟完成签到,获得积分10
8秒前
2111355981完成签到 ,获得积分10
9秒前
科研助理795完成签到,获得积分0
10秒前
舒适涵山完成签到,获得积分0
10秒前
ddd完成签到 ,获得积分10
10秒前
梵强斯完成签到,获得积分10
10秒前
Siren完成签到,获得积分10
10秒前
a_mo完成签到,获得积分10
11秒前
我不爱科研完成签到,获得积分10
11秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1500
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
Analyses de reconnaissances de dettes passées devant les échevins d'Ypres (1249-1291) 600
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8159288
求助须知:如何正确求助?哪些是违规求助? 9554702
关于积分的说明 21720900
捐赠科研通 7748911
什么是DOI,文献DOI怎么找? 3427184
关于科研通互助平台的介绍 2552111
邀请新用户注册赠送积分活动 2471479