正确性
甲骨文公司
计算机科学
障碍物
人工智能
自然语言
自然语言处理
芯(光纤)
机器学习
自然(考古学)
语言模型
作者
S. B. Cho,Stefano Ruberto,Valerio Terragni
标识
DOI:10.1109/icsme64153.2025.00025
摘要
Using Large Language Models (LLMs) to perform Natural Language Processing (NLP) tasks has been becoming increasingly pervasive in recent times. The versatile nature of LLMs makes them applicable to a wide range of such tasks. While the performance of recent LLMs is generally outstanding, several studies have shown that LLMs can often produce incorrect results. Automatically identifying these faulty behaviors is extremely useful for improving the effectiveness of LLMs. One obstacle to this is the limited availability of labeled datasets, necessitating an oracle to determine the correctness of LLM behaviors. Metamorphic Testing (MT) is a popular testing approach that alleviates this oracle problem. At the core of MT are Metamorphic Relations (MRs), defining the relationship between the outputs of related inputs. MT can expose faulty behaviors without the need for explicit oracles (e.g., labeled datasets). This paper presents the most comprehensive study of MT for LLMs to date. We conducted a literature review and collected 191 MRs for NLP tasks. We implemented a representative subset (36 MRs) to conduct a series of experiments with three popular LLMs, running $\sim 560 ~\mathrm{K}$ metamorphic tests. The results shed light on the capabilities and opportunities of MT for LLMs, as well as its limitations.
科研通智能强力驱动
Strongly Powered by AbleSci AI