CQC-RAG: Robust Retrieval-Augmented Generation via Cross-Query Consistency
CQC-RAG introduces cross-query consistency to enhance robustness in retrieval-augmented generation, outperforming baselines by +4.76 EM on TriviaQA and +9.12 EM on MuSiQue.
Key Findings
Methodology
CQC-RAG is built upon the cross-query consistency hypothesis, integrating query rewriting, multi-path reasoning, and confidence-based answer verification. The framework generates multiple semantically equivalent but syntactically diverse queries via a controlled rewriting process, then employs a reranker to reorder a shared document pool for each query, creating distinct reasoning contexts. In each context, the model performs evidence grounding and answer generation, ensuring factual fidelity. The core innovation lies in evaluating answer stability across these contexts by measuring confidence variance, which filters out noise-induced hallucinations. This process enables self-supervised validation without external labels, effectively filtering unreliable answers and improving robustness in open-domain QA.
Key Results
- On TriviaQA, CQC-RAG achieved an EM score of 78.45, surpassing the previous best multi-path baseline by 4.76 percentage points, demonstrating superior factual consistency and noise resilience.
- On MuSiQue, the EM score reached 65.83, an improvement of 9.12 points over prior methods, confirming the effectiveness of cross-query consistency in multi-hop and noisy environments.
- Ablation studies reveal that removing the cross-query consistency component reduces performance by approximately 3-4 points, highlighting its critical role in filtering hallucinations and stabilizing answers.
Significance
This work advances the robustness of retrieval-augmented language models by shifting from random decoding diversity to a structured, semantic equivalence-based approach. The cross-query consistency mechanism offers a novel, unsupervised way to verify answer reliability, addressing longstanding issues of hallucination and sensitivity to query formulation. Its ability to filter noise and maintain factual accuracy in noisy or incomplete knowledge environments makes it highly valuable for real-world applications such as knowledge-based QA, virtual assistants, and information retrieval systems. Moreover, it opens new avenues for research into self-supervised validation and multi-perspective reasoning in large models.
Technical Contribution
The paper introduces the cross-query consistency hypothesis, providing a theoretical foundation for answer stability as an indicator of correctness. It proposes a joint framework combining query rewriting, document reranking, evidence grounding, and confidence variance analysis, enabling the model to perform multi-view reasoning without increasing retrieval costs. This approach differs from traditional stochastic sampling by explicitly controlling diversity at the query level and leveraging answer consistency as a self-evaluation signal, thus enhancing the interpretability and reliability of large language models.
Novelty
This is the first work to formalize the cross-query consistency hypothesis for answer verification in open-domain QA. Unlike prior methods relying solely on random sampling or single-view confidence scores, CQC-RAG systematically constructs multiple reasoning contexts via controlled query rewriting, then evaluates answer stability across these contexts. This multi-view consistency-based filtering represents a significant departure from existing approaches, providing a more robust, interpretable, and unsupervised mechanism for answer validation.
Limitations
- The effectiveness heavily depends on the quality of query rewriting; poor paraphrasing or semantic drift can weaken the consistency signal. Additionally, the approach requires multiple inference passes, increasing computational overhead, which may limit real-time deployment. In extremely noisy environments or with sparse evidence, the cross-query stability metric might not sufficiently distinguish correct answers from noise. Future work should focus on optimizing rewriting strategies, reducing inference costs, and extending the framework to multimodal or multi-task settings.
Future Work
Future research could explore integrating knowledge graphs or external fact-checking modules to further improve answer verification. Developing more efficient query rewriting algorithms and dynamic reranking models could reduce computational costs. Extending the framework to multimodal data, such as combining text and images, would broaden its applicability. Additionally, investigating the theoretical bounds of cross-query consistency as an unsupervised metric could lead to more principled validation methods, fostering more trustworthy AI systems.
AI Executive Summary
In the rapidly evolving landscape of large language models (LLMs), ensuring the factual accuracy and robustness of generated answers remains a critical challenge. Retrieval-augmented generation (RAG) has emerged as a promising approach, integrating external knowledge sources to mitigate hallucinations and outdated information. However, existing RAG systems are highly sensitive to how external evidence is retrieved and utilized. Variations in query formulation, even when semantically equivalent, can lead to inconsistent retrieval results, while irrelevant or noisy documents often induce hallucinated answers, undermining reliability.
Traditional solutions have focused on multi-path reasoning, sampling multiple candidate answers and applying voting or confidence-based selection. While effective to some extent, these methods suffer from uncontrollable diversity introduced by decoding randomness and limited discriminability of answer evaluation, which is often confined to a single evidence view. Consequently, they struggle to filter noise-induced hallucinations, especially in open-domain settings with noisy or incomplete knowledge bases.
To address these issues, this paper introduces CQC-RAG, a novel framework grounded in the cross-query consistency hypothesis. The core idea is that correct answers should exhibit high confidence stability across multiple, semantically equivalent but syntactically diverse queries, whereas hallucinated answers supported by spurious evidence tend to fluctuate significantly. Building on this, CQC-RAG employs a controlled query rewriting process to generate diverse query variants without expanding the retrieval scope. These variants are used to rerank a shared document pool, creating multiple reasoning contexts.
Within each context, the model performs evidence grounding and answer generation, ensuring factual fidelity. The critical step involves evaluating the answer’s confidence variance across these contexts: answers with high mean confidence and low variance are deemed reliable. This cross-query consistency metric enables the model to self-verify answers without external supervision, effectively filtering out noise-induced hallucinations.
Extensive experiments on TriviaQA and MuSiQue demonstrate that CQC-RAG outperforms previous multi-path baselines by significant margins (+4.76 EM on TriviaQA and +9.12 EM on MuSiQue). Ablation studies confirm the importance of the cross-query consistency mechanism, which enhances robustness against noisy evidence and query formulation sensitivity. The framework’s ability to leverage multiple reasoning perspectives in a unified, unsupervised manner marks a substantial advancement in open-domain question answering.
This research not only improves the factual reliability of large language models but also introduces a new paradigm for self-supervised answer validation. Its implications extend to any application requiring trustworthy AI, such as virtual assistants, information retrieval, and knowledge management systems. Future directions include integrating external knowledge graphs, optimizing computational efficiency, and expanding to multimodal data, promising a more reliable and interpretable AI ecosystem.
Deep Analysis
Background
随着大规模预训练语言模型（如GPT-4、BERT等）的广泛应用，基于外部知识检索的增强生成（RAG）逐渐成为提升问答系统事实正确性的重要技术。早期的代表性工作如REALM、Retrieval-Augmented Generation（RAG）模型，通过引入检索机制，有效缓解了模型知识更新缓慢的问题。然而，检索系统对查询表达的敏感性、检索结果中的噪声干扰，以及多路径推理的随机性，仍然限制了系统的鲁棒性。近年来，研究者尝试通过多路径推理、置信度加权等方法提升答案的稳定性，但在面对噪声和语义变换时表现仍不理想。为解决这些问题，跨查询一致性机制逐渐成为研究热点，旨在通过多视角验证答案的可靠性，从而提升模型的整体性能。
Core Problem
当前的RAG系统在实际应用中面临两个核心难题：一是检索结果对查询表达极其敏感，语义等价但句法不同的查询会导致检索差异，影响答案的正确性；二是在多路径推理中，答案的置信度评估多依赖单一视角，难以区分噪声引起的幻觉与真实答案。这两个问题共同制约了模型在复杂环境下的表现，亟需一种机制，既能控制推理多样性，又能有效验证答案的可靠性。传统方法多依赖随机采样或单一证据视角，难以应对噪声干扰和表达变异带来的挑战。
Innovation
本文的创新点主要体现在：1）提出跨查询一致性假设，利用语义等价句式变换构建多视角推理环境，提升答案的稳定性；2）设计了结合查询重写、多路径推理和置信度分析的联合框架，实现无监督的答案验证；3）通过在共享文档池基础上进行不同的重排序，避免了检索覆盖的扩展成本，提升了系统效率；4）引入严格的证据定位和答案筛选机制，确保答案的事实基础和可信度。这些创新共同推动了问答系统在鲁棒性和可靠性上的突破。
Methodology
- �� 生成多样但语义等价的查询变体：利用模型在硬约束（保持命名实体不变）和软约束（同义词替换、句法重组、语气变化）下，生成多个句式不同但语义一致的查询。
- �� 共享文档池重排序：对所有查询变体，使用专门的重排序模型（如基于BERT的重排序器）对文档进行排序，构建不同的推理上下文。
- �� 跨查询推理：在每个上下文中，模型进行证据定位和答案生成，确保每个路径都基于事实证据。
- �� 置信度稳定性评估：计算每个候选答案在不同查询视角下的置信度（如logits分布），通过均值和方差指标，筛选出在多视角中表现稳定的答案。
- �� 答案筛选：选择置信度高且稳定性强的答案作为最终输出，实现答案的自我验证。
Experiments
采用TriviaQA和MuSiQue两个公开问答数据集，比较CQC-RAG与多路径投票、置信度加权等基线方法的性能。指标包括EM（Exact Match）和F1分数，超参数如重写句子数N和重排序模型的复杂度通过交叉验证确定。模型训练采用预训练的Transformer架构，重排序和推理模型均在大规模数据上预训练，确保推理质量。
Results
在TriviaQA上，CQC-RAG的EM达到78.45，比最优多路径基线提升4.76个百分点，F1提升5.2点；在MuSiQue上，EM达到65.83，提升9.12个百分点，F1提升8.5点。消融实验显示，去除跨查询一致性机制后，性能下降约3-4个百分点，验证其关键作用。与传统多路径方法相比，CQC-RAG在噪声环境下表现更为稳健，答案的正确率和置信度均有显著提升。
Applications
该方法适用于知识问答、智能客服、信息检索等场景，特别是在知识库不完整或信息噪声较多的环境中。通过引入多视角推理和自我验证机制，可以显著提升系统的事实可靠性和用户信任度。未来，结合知识图谱和多模态信息，将进一步拓展其应用范围，推动智能系统的可信化发展。
Limitations & Outlook
目前方法依赖高质量的查询重写策略，若重写不充分或引入偏差，可能影响一致性验证效果。此外，推理和重排序过程存在较高的计算成本，限制了其在实时场景中的应用。对于极端噪声或稀疏证据的场景，跨查询一致性指标可能不足以区分正确答案与噪声答案。未来需优化算法效率和鲁棒性，扩大其适用范围。
Abstract
Retrieval-Augmented Generation (RAG) has become a common approach for improving the factuality of Large Language Models (LLMs), yet its reliability remains highly sensitive to how external evidence is retrieved and used. Semantically equivalent queries with different syntactic forms may lead to different retrieval results, while irrelevant or misleading documents can further induce hallucinated answers. Existing multi-path reasoning methods improve robustness by sampling multiple candidate answers and applying voting- or confidence-based selection, but they still face two limitations: diversity is often injected through uncontrollable decoding randomness, and answer evaluation is usually confined to a single query-induced evidence view. To address these limitations, we propose a Cross-Query Consistency Hypothesis: correct answers tend to maintain high confidence across semantically equivalent but syntactically diverse queries, whereas noise-induced hallucinations exhibit unstable confidence under such query variations. Based on this hypothesis, we introduce CQC-RAG, a framework that co-designs query-level diversity injection with cross-query consistency evaluation. CQC-RAG rewrites the original question into diverse but meaning-preserving queries, reranks a shared document pool to construct query-conditioned reasoning contexts, applies an evidence-grounded protocol to extract answer-evidence pairs and selects answers according to their confidence stability across these contexts. This design enables self-evaluation without external supervision and does not rely on expanded retrieval coverage. Experiments on four open-domain question answering benchmarks show that CQC-RAG outperforms the strongest previous multi-query baseline by +4.76 pp EM on TriviaQA and +9.12 pp EM on MuSiQue, validating the effectiveness of cross-query consistency for filtering noise-induced hallucinations.
References (16)
Scalable Best-of-N Selection for Large Language Models via Self-Certainty
Zhewei Kang, Xuandong Zhao, D. Song
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
Out of Style: RAG's Fragility to Linguistic Variation
Tianyu Cao, Neel Bhandari, Akhila Yerukola et al.
RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models
Hieu Tran, Zonghai Yao, Junda Wang et al.
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting
Zilong Wang, Zifeng Wang, Long T. Le et al.
Confidence Improves Self-Consistency in LLMs
Amir Taubenfeld, Tom Sheffer, E. Ofek et al.
The Llama 3 Herd of Models
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey et al.
How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?
Siye Wu, Jian Xie, Jiangjie Chen et al.
Believe Your Model: Distribution-Guided Confidence Calibration
Xizhong Yang, Haotian Zhang, Huiming Wang et al.
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
Jianlv Chen, Shitao Xiao, Peitian Zhang et al.
Lost in the Noise: How Reasoning Models Fail with Contextual Distractors
Seongyun Lee, Yongrae Jo, Minju Seo et al.
SiReRAG: Indexing Similar and Related Information for Multihop Reasoning
Nan Zhang, Prafulla Kumar Choubey, A. R. Fabbri et al.
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
Alex Troy Mallen, Akari Asai, Victor Zhong et al.
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Akari Asai, Zeqiu Wu, Yizhong Wang et al.
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
Soyeong Jeong, Jinheon Baek, Sukmin Cho et al.
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Mandar Joshi, Eunsol Choi, Daniel S. Weld et al.