UniEval-RAG: A Unified End-to-End Evaluation Framework for RAG Systems

Retrieval-Augmented Generation (RAG) reduces hallucinations commonly observed in large language models (LLMs) by integrating external document retrieval. However, existing research lacks a rigorous evaluation of the entire RAG pipeline. Current evaluation frameworks focus on the retrieval and generation stages while neglecting systematic metrics for upstream processes such as text chunking. Furthermore, they have yet to establish a unified evaluation system covering all stages of the RAG pipeline. To address this gap, we introduce UniEval-RAG, a unified evaluation framework that provides end-to-end coverage of the entire RAG pipeline. In addition to evaluating the retrieval and generation stages, this framework incorporates novel metrics, such as keyword density and keyword coverage, to assess the structural coherence and semantic completeness of chunking. Additionally, we introduce a benchmark dataset covering five question answering (QA) scenarios—factual, comparative, analytical, tutorial, and summarization—to support diverse evaluation needs. Experimental results demonstrate that UniEval-RAG effectively identifies performance bottlenecks and offers interpretable insights to guide the optimization of RAG systems.