Less Is More: Elevating RAG via Performance-Driven Context Compression
CORE-RAG employs performance-driven reinforcement learning to compress context to 3%, boosting EM scores by 3.3 points, outperforming full-document baseline.
Key Findings
Methodology
This paper introduces CORE-RAG, a framework combining knowledge distillation for initialization and group relative policy optimization (GRPO) for reinforcement learning. The lightweight compressor is trained end-to-end, directly optimizing for task performance metrics (EM, F1). The compressor, significantly smaller than the downstream LLM, reduces input length drastically while maintaining or improving accuracy. Extensive experiments on datasets like NQ, TriviaQA, HotpotQA, and 2WikiMultihopQA demonstrate that at a compression ratio of 3%, EM scores improve by 3.3 points over full-context baselines, with minimal performance loss. The approach effectively balances efficiency and accuracy, validated across multiple benchmarks.
Key Results
- CORE-RAG achieves a 3.3-point increase in EM scores at only 3% of the original context length, outperforming full-document retrieval and previous compression methods. On NQ and TriviaQA, the method surpasses baseline EM scores by 1.5 to 6.9 points, with a compression ratio as low as 3%. It generalizes well to longer contexts without retraining, maintaining high performance in 10-document scenarios.
- Compared to heuristic and large-model distillation baselines, CORE-RAG demonstrates superior accuracy and efficiency. It reduces input tokens by over 97%, leading to faster inference and lower computational costs, while achieving better or comparable task performance.
- Ablation studies show that knowledge distillation initialization enhances training stability, and reinforcement learning directly aligns compression with task metrics, avoiding performance trade-offs typical of heuristic methods.
Significance
This work addresses the long-standing challenge of balancing context length and task performance in retrieval-augmented generation. By integrating reinforcement learning with knowledge distillation, it introduces a novel, performance-driven compression paradigm that significantly reduces computational overhead without sacrificing accuracy. The method’s ability to generalize across datasets and to longer contexts underscores its potential for real-world deployment in resource-constrained environments. It paves the way for more efficient, scalable, and accurate large language systems, especially in knowledge-intensive applications like question answering and dialogue systems.
Technical Contribution
The main technical innovation lies in formulating context compression as a reinforcement learning problem, where the compressor policy is optimized via group relative policy optimization (GRPO) using task performance as reward. This end-to-end training approach directly aligns compression with task success, unlike prior heuristic or mutual information-based methods. Additionally, the use of knowledge distillation for robust initialization improves training stability. The compressor’s design as a small, efficient model reduces inference costs dramatically. The combination of these elements constitutes a new paradigm for task-aware context compression in large language models.
Novelty
This is the first work to optimize context compression explicitly for task performance using reinforcement learning, rather than relying on proxy heuristics or mutual information objectives. The integration of knowledge distillation for initialization and the use of GRPO for direct reward optimization represent significant departures from existing methods like xRAG, NoiseFilter-IB, or LongLLM-Lingua. The approach achieves high compression ratios with performance gains, establishing a new standard for task-aligned context management.
Limitations
- The method depends on large-scale teacher models for knowledge distillation, which may limit applicability in domain-specific scenarios where such models are unavailable.
- Reinforcement learning training incurs high computational costs, especially for very low compression ratios or highly complex tasks.
- The current evaluation focuses on question answering; performance in more open-ended or generative tasks remains to be validated, and the method may require adaptation for such settings.
Future Work
Future directions include extending the framework to multi-modal tasks, incorporating multi-task learning for broader applicability, and improving sample efficiency of reinforcement learning. Exploring unsupervised or self-supervised initialization strategies could reduce dependency on large teacher models. Additionally, adapting the approach for generative tasks like summarization or dialogue generation, with appropriate reward functions, will broaden its impact. Further research into robustness and domain adaptation will enhance practical deployment in diverse real-world scenarios.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized natural language processing, enabling impressive performance across various tasks. However, their reliance on extensive context inputs poses significant computational challenges, especially in knowledge-intensive applications like question answering. Retrieval-Augmented Generation (RAG) addresses this by retrieving relevant documents and prepending them to prompts, but increasing the number of retrieved documents boosts input length, leading to higher costs and the risk of information loss due to long-context limitations.
Traditional compression methods, such as summarization or heuristic token selection, often trade off task performance for efficiency, relying on proxy objectives that do not directly optimize for downstream accuracy. Recognizing these limitations, the authors propose CORE-RAG, a novel framework that directly aligns context compression with task performance through reinforcement learning. The key innovation is a lightweight compressor trained via a performance-driven reward, using group relative policy optimization (GRPO), which ensures that compressed contexts preserve or enhance task accuracy.
The training process begins with knowledge distillation from a large teacher model, providing a robust initialization. Subsequently, the compressor is fine-tuned to generate summaries that maximize task-specific metrics like Exact Match (EM) and F1 scores. Extensive experiments on datasets such as NQ, TriviaQA, HotpotQA, and 2WikiMultihopQA demonstrate that CORE-RAG achieves a compression ratio of only 3%, yet improves EM scores by 3.3 points compared to full-document retrieval. It outperforms existing compression baselines and generalizes well to longer contexts without retraining.
This approach significantly reduces computational costs while maintaining or improving accuracy, marking a substantial step forward in efficient knowledge retrieval and question answering systems. Its ability to balance efficiency and performance makes it highly promising for deployment in real-world scenarios, including resource-constrained environments. Future work will explore multi-modal extensions, multi-task learning, and adaptation to open-ended generative tasks, further broadening its applicability and impact.
Deep Dive
Plain Language Accessible to non-experts
想象你在准备一场重要的演讲，手头有很多资料，但时间有限。传统的方法是把所有资料都背下来，既费时又容易忘记重点。现在，有个聪明的朋友帮你挑出最关键的内容，只用几句话就能讲清楚。这个朋友就像CORE-RAG，用少量信息就能帮你回答问题，不仅快，还能比用全部资料还准。它通过不断练习，学会只挑最重要的部分，这样既节省时间，又能保证效果。就像你用最少的材料做出最美味的菜一样，效率大大提高，结果还更好。
ELI14 Explained like you're 14
想象你在学校里准备一个演讲，老师给了你很多资料，但你只需要用最重要的部分来讲。以前，你可能会把所有资料都背下来，但那样太累，也容易忘。现在，有个聪明的朋友帮你挑出最关键的内容，只用几句话就能讲清楚。这样既省时间，又能让听众听得懂还记住。CORE-RAG就像这个朋友，用少少的内容帮大模型回答问题，不仅快，还能比用全部资料还准。它通过不断学习，学会挑出最重要的部分，让工作变得更简单、更高效。
Abstract
Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm for improving the timeliness of knowledge updates and the factual accuracy of large language models. However, incorporating a large volume of retrieved documents significantly increases input length, leading to prohibitive computational costs. Existing compression approaches often compromise task performance, primarily due to their reliance on predefined heuristics. These heuristics fail to ensure that the compressed context is conducive to the generation tasks. To address these limitations, we propose CORE-RAG, a novel framework for context compression in RAG systems. CORE eliminates reliance on proxy heuristics through a performance-driven learning framework, which directy utilizes task performance as a feedback signal to iteratively refine the compressor policy. Prior to this optimization process, we incorporate a knowledge distillation phase to initialize the compressor with a robust policy. Extensive experiments demonstrate the superiority of our approach. At a high compression ratio of 3%, CORE not only avoids performance degradation but also improves the average Exact Match (EM) score by 3.3 points compared to using full documents. Our code is available at https://github.com/ziqiangcui/CORE-RAG-ICML26.