医学
背景(考古学)
水准点(测量)
皮肤病科
可靠性(半导体)
诊断准确性
回顾性队列研究
梅德林
医学物理学
队列
公共卫生
诊断试验
临床实习
排名(信息检索)
临床诊断
重症监护医学
循证医学
风险评估
队列研究
临床病史
鉴别诊断
试验预测值
诊断代码
临床判断
疾病严重程度
公共卫生监督
作者
Roy Jiang,Hyunjae Kim,Zhenyue Qin,Morten Lee,Margaret A. MacGibeny,Ailish M. Hanly,Angela Sadlowski,Shanin Chowdhury,Xuguang Ai,Jeff R. Gehlhausen,Qingyu Chen
标识
DOI:10.48550/arxiv.2605.04098
摘要
Multimodal large language models (MLLMs) have demonstrated promise on publicly available dermatology benchmarks. However, benchmark performance may not generalize to real-world dermatologic decision-making. To quantify this benchmark-to-bedside gap, we evaluated four open-weight MLLMs (InternVL-Chat v1.5, LLaVA-Med v1.5, SkinGPT4 and MedGemma-4B-Instruct) and one commercial MLLM (GPT-4.1) across three publicly available dermatology datasets and a retrospective multi-site hospital-based dermatology consultation cohort comprising 5,811 cases and 46,405 clinical images. Models were evaluated on two clinically relevant tasks: differential diagnosis generation and severity-based triage. Diagnostic performance was modest on public datasets and declined substantially in the real-world cohort. On public benchmarks, top-3 diagnostic accuracy reached 26.55% for the best open-weight model and 42.25% for GPT-4.1. On real-world consultation cases using images alone, top-3 diagnostic accuracy fell to 1.50%-13.35% among open-weight models and 24.65% for GPT-4.1. Incorporating clinical context improved performance across all models, increasing top-3 diagnostic accuracy up to 28.75% among open-weight models and 38.93% for GPT-4.1. However, model outputs were highly sensitive to incomplete or erroneous consultation context. For severity-based triage, models achieved moderate sensitivity (above 60%), suggesting potential utility for screening but insufficient reliability for clinical deployment. These findings demonstrate that benchmark performance substantially overestimates the real-world clinical capability of current dermatology MLLMs.
科研通智能强力驱动
Strongly Powered by AbleSci AI