Performance of a Small Language Model Versus a Large Language Model in Answering Glaucoma Frequently Asked Patient Questions: Development and Usability Study

作者
Adriano Cypriano Faneli,Rafael Scherer,Rohit Muralidhar,Marcus Guerreiro-Filho,Luiz Arthur F. Beniz,Verônica Vilasboas-Campos,Douglas Rodrigues da Costa,Alessandro A. Jammal,Felipe Andrade Medeiros
出处
期刊: 卷期号:5: e72101-e72101 被引量:3
标识
DOI:10.2196/72101
摘要

Background: Large language models (LLMs) have been shown to answer patient questions in ophthalmology similar to human experts. However, concerns remain regarding their use, particularly related to patient privacy and potential inaccuracies that could compromise patient safety. Objective: This study aimed to compare the performance of an LLM in answering frequently asked patient questions about glaucoma with that of a small language model (SLM) trained locally on ophthalmology-specific literature. Methods: We compiled 35 frequently asked questions on glaucoma, categorized into 6 domains, including pathogenesis, risk factors, clinical manifestations, diagnosis, treatment and prevention, and prognosis. Each question was posed to both a SLM using a retrieval-augmented generation framework, trained on ophthalmology-specific literature, and to a LLM (ChatGPT 4.0, OpenAI). Three glaucoma specialists from a single institution independently assessed the answers using a 3-tier accuracy rating scale: poor (score=1), borderline (score=2), and good (score=3). Each answer received a quality score ranging from 3 to 9 points based on the sum of ratings from the 3 graders. Readability grade level was assessed using 4 formulas, such as the Flesch-Kincaid Level, the Gunning Fog Index, the Coleman-Liau Index, and the Simple Measure of Gobbledygook Index. Results: The answers from the SLM demonstrated comparable quality with ChatGPT 4.0, scoring mean 7.9 (SD 1.2) and mean 7.4 (SD 1.5), respectively, out of a total of 9 points (P=.13). The accuracy rating was consistent overall and across all 6 glaucoma care domains. Both models provided answers considered unsuitable for health care-related information, as they were difficult for the average layperson to read. Conclusions: Both models generated accurate content, but the answers were considered challenging for the average layperson to understand, making them unsuitable for health care-related information. Given the specialized SLM's comparable performance to the LLM, its high customization potential, lower cost, and ability to operate locally, it presents a viable option for deploying natural language processing in real-world ophthalmology clinical settings.

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
聪慧的冷风完成签到,获得积分10
刚刚
谢逊完成签到,获得积分20
1秒前
crazy发布了新的文献求助10
1秒前
taka发布了新的文献求助10
1秒前
刘雨完成签到 ,获得积分10
1秒前
Mississippiecho完成签到,获得积分0
2秒前
turnsole发布了新的文献求助10
2秒前
Wayne完成签到,获得积分10
2秒前
bo发布了新的文献求助10
2秒前
小陈完成签到,获得积分10
3秒前
3秒前
火鸡味锅巴完成签到 ,获得积分10
3秒前
淡定采波完成签到,获得积分10
3秒前
落后的嚣完成签到,获得积分10
3秒前
GQ完成签到 ,获得积分10
4秒前
Preveil完成签到,获得积分10
4秒前
小周发布了新的文献求助10
4秒前
娜娜子完成签到 ,获得积分10
5秒前
xxszyb完成签到,获得积分10
6秒前
elysia完成签到,获得积分10
6秒前
6秒前
小蘑菇的应助被XY采纳,获得10
6秒前
ZZZ完成签到 ,获得积分10
7秒前
矜持完成签到,获得积分10
7秒前
雪糕发布了新的文献求助10
7秒前
大东东完成签到,获得积分10
7秒前
庾傀斗发布了新的文献求助10
7秒前
陈丰滢完成签到,获得积分10
7秒前
7秒前
颜夜关注了科研通微信公众号
8秒前
望望旺仔牛奶完成签到,获得积分10
9秒前
oldjeff完成签到,获得积分10
9秒前
Verna完成签到,获得积分10
10秒前
熊火焰完成签到,获得积分10
10秒前
zjxnq完成签到 ,获得积分10
10秒前
科研大大完成签到 ,获得积分10
10秒前
jzj完成签到 ,获得积分0
10秒前
风息完成签到,获得积分10
11秒前
鱼大大完成签到,获得积分10
11秒前
浮生若梦完成签到,获得积分10
11秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1500
The World Atlas of Wine 9th edition 1000
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
Analyses de reconnaissances de dettes passées devant les échevins d'Ypres (1249-1291) 600
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8159288
求助须知:如何正确求助?哪些是违规求助? 9554702
关于积分的说明 21720900
捐赠科研通 7748911
什么是DOI,文献DOI怎么找? 3427184
关于科研通互助平台的介绍 2552111
邀请新用户注册赠送积分活动 2471479