Grounded Decoding: Retrieval-Anchored Probability Fusion for Faithful RAG
Proposed Grounded Decoding framework improves factual consistency in RAG without training.
Key Findings
Methodology
The method constructs two matched-prompt distributions: a full RAG distribution conditioned on the query, retrieved documents, and generated prefix, and a retrieval-only distribution conditioned solely on retrieved evidence and the same prefix. The final next-token distribution is derived as the unique solution to a KL-barycenter objective over the probability simplex.
Key Results
- Experiments on ALCE, Natural Questions, and FActScore demonstrate consistent improvements in factual accuracy and citation quality over standard RAG and competitive decoding-time baselines, while maintaining fluency.
- The adaptive weighting scheme provides additional improvements in high-conflict or low-retrieval-quality settings.
- The retrieval distribution is the primary driver of gains, and the KL-barycenter formulation consistently outperforms linear mixtures.
Significance
This research provides a strong and efficient alternative to improve faithfulness in RAG decoding, addressing long-standing challenges in handling external evidence with large language models.
Technical Contribution
Introduced a probability-fusion framework based on a KL-barycenter formulation, providing a closed-form normalized geometric mixture and an adaptive weighting strategy based on distributional disagreement and retriever confidence.
Novelty
First to apply KL-barycenter for probability fusion in RAG decoding, offering clearer probability interpretation compared to existing logit-level intervention methods.
Limitations
- May over-rely on retrieval evidence under low-confidence retrieval conditions, affecting fluency.
- Requires dual-stream decoding process, increasing computational complexity.
Future Work
Future research could explore optimizing the adaptive weighting strategy to enhance applicability and efficiency across different tasks.
AI Executive Summary
As retrieval-augmented generation (RAG) systems scale, ensuring faithful grounding in external evidence becomes increasingly challenging. Large language models may still prioritize parametric knowledge over retrieved information when conflicts arise. We propose a novel training-free decoding framework, Grounded Decoding, designed to improve factual consistency in RAG without modifying model parameters.
Our method constructs two matched-prompt distributions: a full RAG distribution conditioned on the query, retrieved documents, and generated prefix, and a retrieval-only distribution conditioned solely on retrieved evidence and the same prefix. The final next-token distribution is derived as the unique solution to a KL-barycenter objective over the probability simplex, providing a closed-form normalized geometric fusion. This formulation naturally recovers standard RAG when the grounding weight is zero and smoothly shifts probability mass toward retrieved evidence as grounding strength increases.
We further introduce a conflict-aware adaptive weighting scheme that dynamically adjusts grounding based on distributional disagreement and retriever confidence. Experiments demonstrate consistent improvements in factual accuracy and citation quality over standard RAG and competitive decoding-time baselines, while maintaining fluency. Our results indicate that probability-level fusion provides a strong and efficient alternative to logit-level intervention methods for faithful RAG decoding.
Deep Analysis
Background
Retrieval-augmented generation (RAG) has become a standard mechanism for connecting large language models to external knowledge sources. However, these systems still struggle to ensure faithful grounding in retrieved evidence. While retrievers supply passages that can be inspected and cited, generators transform them into fluent responses, but this separation does not guarantee that generated outputs remain fully supported by the provided context.
Core Problem
RAG models face particular challenges in long-form question answering and biography generation, where responses contain multiple atomic facts and only a subset is explicitly supported by the retrieved corpus. The generator may still rely on parametric knowledge, ignore relevant evidence, or produce statements that are plausible but not grounded in the retrieved documents.
Innovation
We propose a novel training-free decoding framework, Grounded Decoding, which constructs two matched-prompt distributions to improve factual consistency in RAG. The final next-token distribution is derived as the unique solution to a KL-barycenter objective over the probability simplex, providing a closed-form normalized geometric fusion.
Methodology
- �� Construct full RAG distribution conditioned on query, retrieved documents, and generated prefix.
- �� Construct retrieval-only distribution conditioned solely on retrieved evidence and the same prefix.
- �� Fuse the two distributions using a KL-barycenter objective.
- �� Introduce adaptive weighting scheme based on distributional disagreement and retriever confidence.
Experiments
We conducted experiments on ALCE, Natural Questions, and FActScore, evaluating Grounded Decoding against standard RAG, CAD, AdaCAD, COIECD, CoCoA, DoLa, and kNN-LM. Experimental settings included matched decoding conditions, retrieved passages, prompt templates, citation format, sampling temperature, truncation strategy, stopping criteria, and maximum generation length.
Results
Experiments show that Grounded Decoding outperforms standard RAG and competitive decoding-time baselines in factual accuracy and citation quality, while maintaining fluency. The adaptive weighting scheme provides additional improvements in high-conflict or low-retrieval-quality settings.
Applications
The method can be directly applied to long-form question answering and biography generation scenarios requiring high factual consistency. It is significant for improving the reliability and citation quality of generated content.
Limitations & Outlook
May over-rely on retrieval evidence under low-confidence retrieval conditions, affecting fluency. The dual-stream decoding process increases computational complexity, potentially impacting real-time applications.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe book (retrieved documents) and your own cooking experience (parametric knowledge). Sometimes, the recipe and your experience may conflict, like the recipe says to add salt, but your experience tells you it's not needed. Grounded Decoding acts like a smart assistant, helping you make decisions at each step by considering both the recipe and your experience, ensuring every dish you make adheres to the recipe while retaining your personal touch.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to answer questions to earn points. There's an assistant in the game that gives you hints (retrieved documents), but you also have your own knowledge (parametric knowledge). Sometimes, the hints and your knowledge conflict, like the hint says the answer is A, but you think it's B. Grounded Decoding is like a super assistant that helps you consider both the hints and your knowledge when answering questions, ensuring your answers fit the hints while keeping your personal style.
Glossary
Grounded Decoding
A training-free decoding framework that improves factual consistency in RAG through probability fusion.
Used to enhance the factual accuracy of generated content.
RAG
A mechanism connecting large language models to external knowledge sources.
Used to generate responses based on retrieved evidence.
KL-barycenter
A probability fusion method that balances distributions through an optimization objective on the probability simplex.
Used to fuse full RAG and retrieval distributions.
Adaptive weighting
A strategy that dynamically adjusts grounding strength based on distributional disagreement and retriever confidence.
Used to enhance the flexibility of grounded decoding.
Retrieval distribution
A distribution conditioned solely on retrieved evidence and generated prefix.
Used for probability fusion with full RAG distribution.
Open Questions Unanswered questions from this research
- 1 How to further optimize adaptive weighting strategy to enhance applicability across tasks?
- 2 How to avoid over-reliance on retrieval evidence under low-confidence conditions?
Applications
Immediate Applications
Long-form QA generation
Enhances factual consistency and citation quality for scenarios requiring high reliability.
Long-term Vision
Real-time applications
Achieve efficient dual-stream decoding in real-time applications, ensuring content accuracy and fluency.
Abstract
As retrieval-augmented generation (RAG) systems scale, it becomes increasingly challenging to ensure faithful grounding in external evidence. Large language models may still prioritize parametric knowledge over retrieved information when conflicts arise. We propose a novel training-free decoding framework, \emph{Grounded Decoding}, designed to improve factual consistency in RAG without modifying model parameters. Unlike standard approaches that rely on a single conditional distribution, our method constructs two matched-prompt distributions at every generation step: (1) a full RAG distribution conditioned on the query, retrieved documents, and generated prefix, and (2) a retrieval-only distribution conditioned solely on retrieved evidence and the same prefix. The final next-token distribution is derived as the unique solution to a KL-barycenter objective over the probability simplex, yielding a normalized geometric fusion of the two distributions.This formulation naturally recovers standard RAG when the grounding weight is zero and smoothly shifts probability mass toward retrieved evidence as grounding strength increases. We further introduce a conflict-aware adaptive weighting scheme that dynamically adjusts grounding based on distributional disagreement and retriever confidence. Experiments on ALCE, Natural Questions, and FActScore demonstrate consistent improvements in factual accuracy and citation quality over standard RAG and competitive decoding-time baselines, while maintaining fluency. Our results indicate that probability-level fusion provides a strong and efficient alternative to logit-level intervention methods for faithful RAG decoding.