计算机科学
标杆管理
核(代数)
软件
软件质量
脆弱性(计算)
数据挖掘
软件工程
Linux内核
数据建模
建模语言
可靠性工程
语言模型
人工智能
核密度估计
实时计算
软件系统
作者
Arastoo Zibaeirad,Rodrigo Pato Nogueira,Marco Vieira
标识
DOI:10.1109/tr.2026.3716597
摘要
Large language models (LLMs) demonstrate strong capabilities in code-related tasks, however their effectiveness in software vulnerability detection (SVD) remains poorly understood due to inadequate evaluation frameworks. Existing benchmarks suffer from training data contamination, isolated function evaluation without cross-component context, lack of strict vulnerable-patched pairing, and binary classification without hierarchy-aware common weakness enumeration (CWE) assessment, which prevents a reliable measurement of security reasoning versus pattern matching on leaked data. We introduce LLMKernelBench, a rigorous benchmark comprising 417 real-world Linux kernel vulnerabilities across 74 CWE types, split into a primary benchmark dataset (PBD; 314 samples,$\leq$2024) and a leakage free dataset (LFD; 103 samples, 2025 post-cutoff), with context-aware evaluation at three granularity levels and hierarchy-aware metrics quantifying semantic proximity in misclassifications. We evaluate seven LLM spanning code-specialized and general-purpose architectures. Binary vulnerability detection is near-random ($\sim\!\! 50\%$accuracy) and strongly biased: some models label$>75\%$of samples as vulnerable, while others mostly label them as nonvulnerable. CWE prediction is effectively unusable, with an average Top-1 accuracy of 1.4% (best: 3.3%) and a 12.3% hierarchy proximity score, providing little reliable exact or taxonomy-level signal. On the leakage-free 2025 split, binary accuracy remains near-random and robustness to multifile abstraction is model-specific rather than tied to specialization, with code-specialized and general-purpose models degrading by 2.6% and 4.5% on average, respectively. The micro-to-macro CWE-accuracy gap is larger on LFD (6.6 points) than on PBD (0.9 points), which is consistent with sensitivity to class frequency but does not identify an internal model mechanism.
科研通智能强力驱动
Strongly Powered by AbleSci AI