From hallucinations to hazards: benchmarking LLMs for hazard analysis in safety-critical systems

标杆管理 公制(单位) 一致性(知识库) 危害 危害分析 风险分析(工程) 蓝图 水准点(测量) 毒物控制 计算机科学 风险评估 工程类 鉴定(生物学) 风险管理 质量(理念) 质量保证 人为因素与人体工程学 安全管理体系 职业安全与健康 数据科学 管理科学 概念框架 绩效衡量
作者
Ιoannis M. Dokas
出处
期刊:Safety Science [Elsevier BV]
卷期号:194: 107056-107056 被引量:42
标识
DOI:10.1016/j.ssci.2025.107056
摘要

• Identifies gaps in current LLM benchmarks for hazard analysis. • Proposes a conceptual framework for creating specialized, domain-specific benchmarks. • Presents a pilot validation study with quantitative metrics on LLM performance. • Introduces Performance Consistency as a critical, new metric for evaluating LLMs in hazard analysis. • Issues a call for interdisciplinary collaboration on LLM benchmarks for safety critical systems. Integrating Large Language Models (LLMs) into safety–critical domains presents both promising opportunities and significant challenges for hazard analysis. While these AI systems demonstrate impressive capabilities in natural language understanding and generation, their reliability, accuracy, and trustworthiness in safety contexts remain critical concerns. This paper maps the current landscape of LLM benchmarks via a scoping review, categorizing them based on their primary focus and evaluating their applicability to safety–critical hazard analysis. The review reveals significant limitations in existing benchmarks, including inadequate coverage of safety-specific knowledge, limited evaluation of causal reasoning in technical contexts, the absence of regulatory compliance assessment, insufficient risk analysis capabilities, and minimal uncertainty handling evaluation. To address these gaps, a methodological blueprint for future benchmark development is proposed, tailored to hazard analysis in safety–critical systems. To test this blueprint, a pilot study was conducted across nine safety–critical scenarios, executed over three runs to measure performance consistency over time. This analysis of performance consistency across three evaluation runs provided initial evidence of significant volatility. While the model’s success rate in generating a response was stable, its analytical quality was inconsistent. Hazard identification scores varied between runs, and causal reasoning performance was consistently poor and unpredictable. This evidence of inconsistent analytical quality, even from a model of the same version, highlights a significant potential challenge for safety assurance.
最长约 10秒,即可获得该文献文件

科研通智能强力驱动
Strongly Powered by AbleSci AI
科研通是完全免费的文献互助平台,具备全网最快的应助速度,最高的求助完成率。 对每一个文献求助,科研通都将尽心尽力,给求助人一个满意的交代。
实时播报
suans发布了新的文献求助10
1秒前
1秒前
起风了完成签到,获得积分10
1秒前
单薄傲易发布了新的文献求助10
2秒前
2秒前
隐形曼青的应助被千里洛采纳,获得10
2秒前
Zoey完成签到,获得积分10
3秒前
4秒前
三国时代发布了新的文献求助10
5秒前
6秒前
7秒前
小张医生完成签到,获得积分10
7秒前
8秒前
Jazz33333333完成签到,获得积分10
8秒前
CoCo发布了新的文献求助10
8秒前
8秒前
所所的应助被Zoey采纳,获得10
9秒前
endlessloop发布了新的文献求助10
11秒前
8R60d8的应助被dingo采纳,获得10
11秒前
李健的应助被合适忆枫采纳,获得20
11秒前
12秒前
欣慰的疾完成签到,获得积分20
12秒前
淡然冬灵发布了新的文献求助10
12秒前
JamesPei的应助被白日梦采纳,获得10
14秒前
14秒前
15秒前
风吹麦浪的应助被奋斗断秋采纳,获得10
16秒前
16秒前
endlessloop完成签到,获得积分10
16秒前
Jerry发布了新的文献求助10
17秒前
和谐的数据线完成签到 ,获得积分10
17秒前
18秒前
科研通AI6.3的应助被小陈采纳,获得10
18秒前
欣慰的疾发布了新的文献求助10
19秒前
20秒前
youbei发布了新的文献求助10
21秒前
千里洛发布了新的文献求助10
21秒前
共享精神的应助被美丽采纳,获得10
21秒前
科研通AI6.3的应助被眼药水采纳,获得10
23秒前
大个的应助被开心德地采纳,获得10
23秒前
高分求助中
(应助此贴封号)【重要!!请各用户(尤其是新用户)详细阅读】【科研通的精品贴汇总】 10000
拉伸高分子熔体中缠结行为的模拟和小角中子散射研究 1700
The World Atlas of Wine 9th edition 1000
Sex, Aging, and Death in a Medieval Medical Compendium: Trinity College Cambridge MS R.14.52, Its Texts, Language, and Scribe 500
高分子基功能界面水蒸发材料与应用研究 500
外场增强的水汽发电研究 500
基于结构调控的功能化水诱导发电材料与器件研究 500
热门求助领域 (近24小时)
化学 材料科学 医学 生物 纳米技术 计算机科学 工程类 有机化学 化学工程 生物化学 物理 复合材料 内科学 催化作用 光电子学 人工智能 细胞生物学 无机化学 物理化学 数学
热门帖子
关注 科研通微信公众号,转发送积分 8189137
求助须知:如何正确求助?哪些是违规求助? 9576601
关于积分的说明 21848766
捐赠科研通 7787145
什么是DOI,文献DOI怎么找? 3435490
关于科研通互助平台的介绍 2559732
邀请新用户注册赠送积分活动 2482462