CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG
CoinRAG employs fine-grained nugget KV cache reuse to enable efficient long-context RAG, achieving a 5.3% relative F1 improvement under 100ms latency.
Key Findings
Methodology
This paper introduces the CoinRAG framework, which offline extracts semantic units called 'nuggets' from text chunks. During online inference, a two-stage retrieval process identifies query-relevant nuggets, and their precomputed KV cache slices are assembled with chunk-level context using position offset mechanisms. The approach replaces full chunk encoding with a more compact, semantically relevant cache composition, significantly reducing inference latency and resource consumption. The system incorporates nugget-aware fine-tuning to align training with inference procedures, ensuring high accuracy. Extensive experiments on LongBench multi-hop QA datasets demonstrate that CoinRAG outperforms baseline methods, achieving an average 5.3% F1 gain within the 100ms latency budget, while also reducing context length by 1.84×.
Key Results
- In multi-hop QA tasks from LongBench, CoinRAG maintains superior accuracy under strict latency constraints, with a 5.3% F1 score increase over standard RAG, and reduces context length from 723 tokens to 387 tokens on average. Ablation studies confirm that fine-grained nugget KV slicing outperforms full chunk encoding, and offline extraction surpasses online generation in efficiency and accuracy. The position offset mechanism effectively mitigates misalignment issues during cache concatenation, further boosting performance.
- When no latency constraints are applied, CoinRAG still surpasses two strong baselines (TurboRAG and KVLink), with an average F1 improvement of 5.2%. The method demonstrates robust generalization across datasets, including HotpotQA, 2WikiMQA, and MuSiQue, with consistent performance gains. Pareto frontiers illustrate that CoinRAG achieves higher accuracy with substantially shorter context lengths and lower GPU memory usage, making it suitable for real-time deployment.
- The experimental results highlight that fine-grained, query-specific cache reuse combined with offline nugget extraction provides a scalable solution for long-context question answering, especially under strict latency and resource constraints. The approach balances semantic richness and computational efficiency, setting a new benchmark for retrieval-augmented generation systems.
Significance
This work addresses the critical challenge of balancing inference efficiency and answer accuracy in retrieval-augmented systems handling long texts. By introducing a novel nugget-based KV cache reuse strategy, CoinRAG reduces computational redundancy and memory footprint, enabling real-time responses in interactive applications. Its ability to maintain high factual accuracy while significantly lowering latency paves the way for deploying large language models in latency-sensitive environments like customer service, virtual assistants, and knowledge bases. Moreover, the framework's modular design facilitates integration with existing retrieval and generation architectures, fostering broader adoption of efficient long-context NLP solutions. This research marks a significant step toward scalable, resource-efficient, knowledge-enhanced AI systems.
Technical Contribution
The paper's main technical innovations include: (1) offline extraction of semantic nuggets from text chunks using LLM prompting, which ensures high-quality, contextually grounded units; (2) a two-stage retrieval process that first narrows down candidate chunks, then ranks nuggets based on similarity, improving relevance filtering; (3) a position offset mechanism that aligns sliced KV caches from different chunks, preserving semantic coherence during concatenation; and (4) nugget-aware fine-tuning that trains the model on simulated inference scenarios to optimize cache integration. These components collectively enable a cache composition strategy that is both compact and semantically rich, reducing latency and resource usage without sacrificing accuracy.
Novelty
Unlike prior cache-augmented methods that operate at the chunk level or rely on online nugget generation, CoinRAG precomputes and stores fine-grained semantic units offline, then dynamically assembles relevant slices during inference. Its two-stage retrieval and position offset techniques address the challenge of aligning heterogeneous cache segments, ensuring semantic integrity. This approach introduces a new granularity in cache reuse, bridging the gap between coarse chunk-level caching and fine-grained online nugget generation, thus achieving superior efficiency and accuracy. To the best of our knowledge, this is the first framework to combine offline nugget extraction with dynamic, query-driven cache assembly for long-context RAG.
Limitations
- While effective under moderate-length texts, the method's reliance on offline nugget extraction may struggle with highly dynamic or domain-specific knowledge bases, requiring frequent reprocessing. In scenarios with extremely sparse or noisy nuggets, relevant information might be missed, impacting answer quality.
- The offline extraction process depends on preset similarity thresholds and chunk segmentation strategies, which may need domain-specific tuning, limiting out-of-the-box applicability across diverse tasks.
- The nugget-aware fine-tuning adds complexity and training overhead, and may require retraining when deploying in new domains or with updated knowledge sources. Additionally, the method's performance could degrade if the query relevance estimation is inaccurate.
Future Work
Future directions include developing adaptive nugget extraction algorithms that dynamically adjust granularity based on context, integrating multi-modal retrieval for richer knowledge representations, and exploring continual learning techniques to update nuggets without retraining from scratch. Extending the framework to multi-modal long documents, such as videos or images with text, could broaden its applicability. Further research might also focus on optimizing the position offset mechanism for even better alignment accuracy and exploring end-to-end training strategies that jointly optimize nugget extraction, retrieval, and generation for holistic performance improvements.
AI Executive Summary
In recent years, large language models (LLMs) have revolutionized natural language processing, enabling sophisticated applications like question answering, summarization, and dialogue generation. However, their deployment in real-time, interactive environments faces significant challenges, especially when handling long contexts. Traditional retrieval-augmented generation (RAG) systems enhance LLMs by integrating external knowledge, but they often rely on encoding entire documents or large chunks during inference, which incurs high computational costs and latency.
This bottleneck becomes critical in scenarios demanding response times under 100 milliseconds, such as customer support chatbots or virtual assistants. Existing solutions like cache-augmented generation (CAG) precompute and reuse key-value (KV) caches at the chunk level, reducing some redundancy but still suffering from coarse granularity, noise, and limited semantic relevance. Moreover, online nugget generation methods, while more fine-grained, introduce high latency due to expensive LLM calls.
Addressing these issues, the paper introduces CoinRAG, a novel framework that combines offline extraction of semantic 'nuggets' with query-driven retrieval and cache assembly. The core idea is to precompute and store small, meaningful text spans—nuggets—during offline processing. During inference, a two-stage retrieval process first narrows down candidate chunks, then selects the most relevant nuggets based on similarity scores. These nuggets’ KV caches are sliced from precomputed full-chunk caches using position offset mechanisms, ensuring semantic coherence and contextual grounding.
This approach effectively balances the need for semantic richness and computational efficiency. By assembling only the most relevant, contextually grounded cache slices, CoinRAG significantly reduces the amount of data processed in real-time, lowering latency and memory footprint. The system incorporates nugget-aware fine-tuning, aligning training with inference procedures, further boosting accuracy.
Experimental results on LongBench multi-hop QA datasets demonstrate that CoinRAG outperforms traditional RAG and other cache reuse baselines, achieving an average 5.3% F1 score improvement within the 100ms latency constraint. It also maintains high accuracy even without latency limits, showcasing robustness and scalability. The method reduces context length by 1.84×, translating into substantial hardware savings and higher throughput.
This research advances the state-of-the-art in efficient long-context retrieval and generation, providing a practical solution for deploying knowledge-intensive AI systems in latency-sensitive applications. Its modular design and strong empirical performance suggest broad applicability across NLP tasks requiring long document understanding, multi-hop reasoning, and real-time interaction.
Looking ahead, future work could explore adaptive nugget extraction, multi-modal extensions, and continual knowledge updates, further enhancing the framework’s flexibility and effectiveness. Overall, CoinRAG marks a significant step toward scalable, resource-efficient, and accurate long-context AI systems, promising transformative impacts on industry and academia alike.
Deep Dive
Abstract
Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise still remain in the coarse-grained chunks. This paper optimizes the Pareto frontier under low prefill latency constraints while maximizing accuracy by proposing CoinRAG (Contextualized Information Nugget KV Cache Reuse for Long-Context RAG). The name metaphorically reflects our core mechanism: much like assembling small tokens (or "coins") to accumulate a larger value, CoinRAG compositionally reuses offline-computed, fine-grained nugget caches to form a learned contextual representation efficiently in a more semantically relevant but compact manner. Specifically, instead of full-chunk encoding, CoinRAG identifies query-relevant semantic units within retrieved chunks through two-stage retrieval and seamlessly assembles their sliced KV representations with a chunk-level context. Extensive evaluations on LongBench multi-hop question answering tasks demonstrate that CoinRAG significantly reduces operational costs and outperforms the other baselines with a new Pareto frontier and an average 5.3% relative improvement in answer quality (F1) under a standard fast prefill latency budget.
References (20)
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Yushi Bai, Xin Lv, Jiajie Zhang et al.
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Ronak Pradeep, Nandan Thakur, Shivani Upadhyay et al.
RoFormer: Enhanced Transformer with Rotary Position Embedding
Jianlin Su, Yu Lu, Shengfeng Pan et al.
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
H. Trivedi, Niranjan Balasubramanian, Tushar Khot et al.
KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
Jingbo Yang, Bairu Hou, Wei Wei et al.
Enabling Large Language Models to Generate Text with Citations
Tianyu Gao, Howard Yen, Jiatong Yu et al.
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
A. Gujarati, Reza Karimi, Safya Alzayat et al.
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
In Gim, Guojun Chen, Seung-seob Lee et al.
Block-Attention for Efficient Prefilling
Dongyang Ma, Yan Wang, Tian Lan
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Jiayi Yao, Hanchen Li, Yuhan Liu et al.
Retrieval Augmentation Reduces Hallucination in Conversation
Kurt Shuster, Spencer Poff, Moya Chen et al.
CoQA: A Conversational Question Answering Challenge
Siva Reddy, Danqi Chen, Christopher D. Manning
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Xanh Ho, A. Nguyen, Saku Sugawara et al.
Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
Brian J. Chan, Chao-Ting Chen, Jui-Hung Cheng et al.
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al.
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al.
A Survey on Large Language Model Acceleration based on KV Cache Management
Haoyang Li, Yiming Li, Anxin Tian et al.
Clipper: A Low-Latency Online Prediction Serving System
D. Crankshaw, Xin Wang, Giulio Zhou et al.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandara Piktus et al.
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text
Songshuo Lu, Hua Wang, Yu Rong et al.