Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability
Q-CARE introduces query coverage and claim verifiability for reference-free, fine-grained RAG evaluation, outperforming existing metrics.
Key Findings
Methodology
Q-CARE employs a three-stage process: first, decomposing queries into sub-questions and answers into atomic claims; second, using large language models (e.g., Qwen-30B) to align retrieved documents, sub-questions, and claims, assessing coverage and support; third, aggregating these signals into retrieval and generation metrics, including coverage-aware Precision@k, nDCG@k, and claim-level metrics (completeness, conciseness, verifiability). The framework is fully automatic, supports diverse query types, and is validated on eight datasets, showing higher correlation with human judgments than prior metrics.
Key Results
- Q-CARE achieves a Pearson correlation of up to 0.69 with human judgments across datasets, outperforming RAGEval (0.29) and RAGChecker (0.32). Coverage metrics (C-Prec@k, C-nDCG@k) show stable performance across query types, especially in open-ended QA. Claim-level metrics effectively capture factuality and completeness, with larger models like Qwen-30B providing better evaluation consistency. The framework demonstrates robustness and scalability, validating its practical utility.
Significance
This work advances the evaluation of knowledge-augmented models by providing a unified, reference-free, fine-grained assessment method. It addresses the limitations of traditional binary relevance and reference-based metrics, enabling more accurate and scalable benchmarking across diverse tasks. The approach enhances model transparency, supports continuous improvement, and facilitates deployment in real-world applications such as open-domain QA, content moderation, and knowledge verification, ultimately pushing forward the development of reliable AI systems.
Technical Contribution
The paper introduces a novel decomposition-based evaluation framework that combines query sub-question and answer atomic claim decomposition with large model-based alignment. It innovates by replacing binary relevance with soft, coverage-aware relevance scores, enabling continuous and more informative metrics. The three-stage process—decomposition, alignment, aggregation—provides a comprehensive, automated, and fine-grained evaluation pipeline that surpasses existing methods in both scope and precision. The framework is flexible, model-agnostic, and validated across multiple datasets and models, demonstrating significant improvements over state-of-the-art benchmarks.
Novelty
This is the first framework to unify query coverage and claim verifiability into a single, reference-free evaluation paradigm applicable to both close- and open-ended queries. It leverages large language models for automatic decomposition and alignment, introducing coverage-aware relevance metrics that move beyond binary labels. Unlike prior work limited to either coarse metrics or reference-dependent fine-grained evaluation, Q-CARE provides a scalable, universal solution for comprehensive RAG assessment, filling a critical gap in current evaluation methodologies.
Limitations
- The framework relies heavily on large pre-trained models like Qwen-30B, which entail high computational costs, limiting real-time deployment. Its performance in multi-hop, multi-modal, or highly complex reasoning scenarios remains to be fully validated. Additionally, in cases of extreme out-of-distribution queries or hallucinated responses, the alignment accuracy may decline, affecting evaluation reliability. Future work should focus on efficiency improvements and multi-modal extensions.
Future Work
Future directions include integrating multi-modal data (images, audio) to evaluate multi-source models, optimizing model efficiency for real-time deployment, and developing adaptive, user-feedback-driven evaluation metrics. Expanding the benchmark to cover more languages and domains will also be prioritized, aiming to establish a comprehensive, standardized evaluation ecosystem for knowledge-enhanced AI systems.
AI Executive Summary
The rapid development of large language models (LLMs) like GPT-4 and Qwen-30B has revolutionized natural language understanding and generation. However, evaluating their factual correctness and knowledge grounding remains a significant challenge. Traditional metrics such as EM and ROUGE depend heavily on reference answers, which are often unavailable or insufficient for diverse, open-ended queries. This gap hampers reliable benchmarking and hinders progress toward trustworthy AI.
To address this, the paper introduces Q-CARE, a novel evaluation framework based on query coverage and claim verifiability. It decomposes complex queries into sub-questions and answers into atomic claims, enabling fine-grained assessment without relying on references. The core innovation lies in automatic alignment using large language models, which evaluate how well retrieved documents support each sub-question and claim. This process produces coverage-aware retrieval metrics (C-Prec@k, C-nDCG@k) and generation metrics (completeness, conciseness, verifiability), providing a comprehensive evaluation across query types.
Extensive experiments on eight datasets demonstrate that Q-CARE correlates more strongly with human judgments than existing metrics, achieving a correlation coefficient up to 0.69. Its robustness across models and query types highlights its potential as a standard evaluation tool for knowledge-grounded AI systems. The framework's flexibility allows adaptation to multi-modal data and broader domains, promising a future where AI evaluation is more transparent, scalable, and aligned with human expectations.
Despite its strengths, the approach faces challenges such as high computational costs due to reliance on large models and limited validation in highly complex reasoning scenarios. Future work will focus on efficiency improvements, multi-modal integration, and expanding benchmark coverage, aiming to foster trustworthy and reliable AI systems in real-world applications.
Deep Dive
Limitations & Outlook
What gaps remain?
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂的目标是生产高质量的产品。每个工人（模型）都要完成不同的任务，比如组装、检查、包装。为了确保每个产品都符合标准，你会逐个检查每个环节，而不是只看最终成品。传统方法就像只看最后的产品是否完美，但有时候，产品外表看起来不错，但内部可能有问题。现在，Q-CARE就像是一个智能检测系统，它会把每个环节拆开，逐个检查每个部分是否都符合要求。它会问每个工人：“你这个部分是不是做得完整？有支持你的证据吗？”通过这样逐步检查，能更准确地判断整个生产线的质量。这种方法比只看成品更细致，也更可靠，能帮助工厂不断改进，生产出更好的产品。
ELI14 Explained like you're 14
想象你在学校做科学实验，你需要写一份报告告诉老师你学到了什么。以前，你只需要把结果写出来，老师就知道你懂了。但有时候，老师还想知道你是怎么知道这些的。于是，你开始把每个步骤都写得更详细，比如你用的资料、做的实验、得到的结论。这样，老师就能更清楚你是不是真正理解了。Q-CARE就像这个过程，它把你的报告拆成很多小部分：每个问题的答案、用到的证据、每个步骤的解释。然后，它会检查每个部分是不是都支持你的结论。这样一来，不仅能知道你做得对不对，还能知道你是不是用对了资料、每个答案是不是都靠谱。这种方法让评估变得更细、更准，也更公平。
Glossary
Retrieval-Augmented Generation (检索增强生成)
结合检索系统和生成模型，通过检索相关资料支持生成内容，提升回答的事实性。
描述如何利用检索信息增强生成模型的事实性和可靠性。
Query Decomposition (查询分解)
将复杂查询拆解为多个子问句，以便更细粒度理解和处理信息需求。
在Q-CARE中，将复杂查询转化为多个简化子任务。
Claim Verifiability (声明可验证性)
判断生成的声明是否有检索证据支持，确保内容的真实性。
核心指标之一，用于评价回答的事实支持程度。
Coverage-aware Metrics (覆盖率感知指标)
基于子问句覆盖情况设计的检索指标，反映检索内容的全面性。
替代传统二元相关标签，提升评估的连续性和鲁棒性。
Atomic Claims (原子声明)
回答中的最小、可验证的陈述单元，便于细粒度评估。
在回答分解阶段生成，用于后续对齐和验证。
Open Questions Unanswered questions from this research
- 1 如何进一步降低大模型在多模态、多源信息场景中的评估成本和复杂度，仍需探索多模态融合与高效对齐技术。
- 2 在极端偏离常识或多步骤推理的复杂场景中，指标的鲁棒性和准确性仍有待验证。
Abstract
Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework. Code and data are publicly available at https://github.com/DISL-Lab/Q-CaRE-COLM-26.