Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study

计算机科学 印为红字的 水准点(测量) 自然语言处理 医学物理学 审计 人工智能 可用性 机器学习 语言模型 答辩人 裁决 听力学家 文档 语音识别 清晰 最佳实践 临床实习 心理学 能力(人力资源) 学习曲线 下垂 标杆管理 质量得分 梅德林 医学
作者
Linkai Li,Changgeng Mo,Haoshuai Zhou,Hanlin Yu,Congxi Lu,Shangqiguo Wang,Varsha Mysore Athreya,Matthew B. Fitzgerald,Shan Xiang Wang
出处
期刊:Journal of Medical Internet Research [JMIR Publications]
卷期号:28: e94755-e94755 被引量:2
标识
DOI:10.2196/94755
摘要

Background: Large language models (LLMs) are increasingly being explored for clinical decision support, but their performance in audiology has not been systematically benchmarked using clinically grounded case materials and rubric-based safety evaluations. Objective: This study aimed to develop and evaluate AUDIOLOGYBENCH, a 3-tier benchmark for characterizing frontier LLM capability in clinical audiology along (1) curated domain knowledge, (2) literature-derived evidence, and (3) clinical reasoning under multimodal case input, with an explicit human audit of the automated adjudicator on the primary end point. Methods: The benchmark comprises 3139 objective items from educational resources, 3175 research article-derived items from peer-reviewed articles published between 2015 and 2025, and 67 multimodal clinical case studies graded against a standardized A-F rubric with 6 prespecified critical-error types that cap scores at D or F. Eight models were evaluated on the educational objective items: 4 frontier multimodal models (Gemini 2.5 Pro, Grok 4, OpenAI O3, and Claude Sonnet 4 Thinking) were evaluated on the research article-derived items, and on 804 case study evaluations. Adjudication used Gemini 2.5 Pro (objective and research-derived items) and Claude Opus 4.5 (case studies). The case study adjudicator was independently audited against PhD-level audiologist consensus on blinded subsamples, supplemented by a post-stratified human-calibrated sensitivity analysis. Results: ≥0.679). Interreviewer reliability between audiologists was high (quadratic-weighted κ of 0.78 and 0.85 across the 80-item and 50-item audits, respectively). When 2 audiologists regraded all 80 model Q1 responses with the diagnostic images available, the adjudicator's per-item Q1 labels diverged from human judgment (κ=0.05; overflagging; sensitivity: 19/26, 73%; positive predictive value: 19/53, 36%), yet its reweighted Q1 critical-error rate (36.2%) was broadly consistent with the image-grounded human estimates (28%-34%), suggesting no systematic inflation of the headline rate. The principal Q3>{Q1, Q2} ranking was preserved under post-stratified human calibration. Web-style multiple-choice items showed ceiling effects (>95% accuracy); short-answer prompts remained challenging (best 30%). Conclusions: Current frontier LLMs show strong recommendation generation but substantial limitations in audiometric numerical interpretation that are shared across models and that an automated adjudicator partially miscalibrated at the per-item level. AUDIOLOGYBENCH characterizes capability boundaries rather than certifying clinical readiness. Deployment of LLM-assisted audiology workflows requires structured human verification of all numerical findings and awareness of fabrication and severity misclassification failure modes documented here.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
研友_VZG7GZ的应助被KTLJ采纳,获得10
刚刚
都会有完成签到,获得积分20
刚刚
半亩月光发布了新的文献求助10
刚刚
XiYang发布了新的文献求助10
1秒前
7788999完成签到,获得积分10
1秒前
大气摩托完成签到,获得积分10
1秒前
关耳完成签到 ,获得积分10
2秒前
2秒前
安安的应助被知了采纳,获得10
2秒前
大巧若拙发布了新的文献求助10
2秒前
猎人日记完成签到,获得积分10
2秒前
Jasper的应助被光亮的书文采纳,获得10
2秒前
ttttt完成签到,获得积分10
3秒前
3秒前
kevin231完成签到,获得积分10
4秒前
4秒前
Ava的应助被Lu采纳,获得10
4秒前
NexusExplorer的应助被都会有采纳,获得10
5秒前
脑洞疼的应助被lhz采纳,获得10
5秒前
CipherSage的应助被加油采纳,获得10
6秒前
华仔的应助被加油采纳,获得10
6秒前
李健的应助被奋斗的凌青采纳,获得30
6秒前
CarryLJR发布了新的文献求助30
6秒前
科目三的应助被温暖砖头采纳,获得10
7秒前
7秒前
美满怀绿发布了新的文献求助10
7秒前
CipherSage的应助被LHM采纳,获得10
7秒前
传奇3的应助被LHM采纳,获得10
7秒前
大巧若拙完成签到,获得积分10
8秒前
8秒前
Xlinging完成签到,获得积分10
8秒前
orixero的应助被拼搏的似狮采纳,获得10
9秒前
ohyeah完成签到,获得积分10
9秒前
murphy完成签到,获得积分10
9秒前
9秒前
10秒前
陈旻完成签到,获得积分10
10秒前
迨你个迨迨完成签到,获得积分10
10秒前
gem完成签到 ,获得积分10
10秒前
12秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
奈特基础组织学:配套组织病理学(第4版)(Netter's Essential Histology With Correlated Histopathology) 800
John McDowells Bildungstheorie 800
八旗制度の研究 600
plantTFDB 508
Resource Recovery Handbook 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8175556
求助须知:如何正确求助?哪些是违规求助? 9567240
关于积分的说明 21791502
捐赠科研通 7770174
什么是DOI,文献DOI怎么找? 3431868
关于科研通互助平台的介绍 2556244
邀请新用户注册赠送积分活动 2477299