DelphiAgent: A trustworthy multi-agent verification framework for automated fact verification

Large Language Models (LLMs) have been investigated for many reasoning-intensive tasks including fact verification and exhibited outstanding performance via coupling LLM’s internal and external knowledge. However, non-agentic LLM-based methods produce responses based on direct prompts in an one-off manner, suffering from challenges in factuality and hallucinations . In this paper, we propose DelphiAgent, an innovative agentic framework for trustworthy fact-checking that employs multiple LLMs to emulate the workflow of the Delphi method , aiming at enhancing transparency in the decision-making procedure and mitigating hallucinations when generating justifications. This is implemented through a duel-system framework that integrates the evidence mining module and the Delphi decision-making module. The evidence mining module extracts evidence from raw uncensored reports and refines evidence, ensuring the provision of instructive rationales for the subsequent module. Meanwhile, drawing inspiration from the Delphi method , the decision-making module devises multiple LLM-based agents with distinct personalities to make factuality judgments individually based on the claim and its verified evidence, and reaches a consensus through multiple rounds of feedback and synthesis. The experimental findings from two challenging datasets indicate that DelphiAgent not only surpasses current LLM-based approaches but also is on par with state-of-the-art LLM-enhanced supervised baselines without necessitating a training regime, with macF1 improvements reaching up to 6.84% on RAWFC and comparable performance on LIAR-RAW. Furthermore, the generated justifications throughout the workflow underscore the trustworthiness of our proposed framework. The official implementation of this paper is available at https://github.com/zjfgh2015/DelphiAgent . • Non-agentic LLMs suffer from challenges in factuality and hallucinations. • DelphiAgent employs multiple LLMs to emulate the Delphi method . • DelphiAgent enhances transparency and mitigates hallucinations. • DelphiAgent competes SOTA baselines without necessitating a training regime.