An experimental study of KV cache reuse strategies in chunk-level caching systems

The evolution of large language models (LLMs) like GPT and BERT has driven significant progress in NLP. To address inference latency, techniques such as prefix caching and chunk-level caching (CLC) have been developed, enabling reuse of key-value (KV) caches for common prompt segments. Early works focused on static prefix reuse, but their effectiveness diminishes with prompt variability. Recent innovations include recomputation methods (Cacheblend, EPIC) and attention reshaping (APE, SEL), aimed at approximating full attention behavior while reducing computation. Despite these advances, the core challenge remains: how to efficiently recover cross-chunk attention dependencies that are vital for semantic coherence and reasoning accuracy. This paper builds upon these foundations, systematically evaluating their limitations and exploring synergistic combinations to enhance inference quality.

The main challenge in chunk-level caching is the neglect of cross-chunk attention, which leads to semantic dissonance and reduced accuracy in model outputs. While caching individual chunks reduces latency, it fails to capture dependencies across chunks, especially in multi-hop reasoning tasks requiring integrated context. Existing solutions like recomputation and attention reshaping partially address these issues but are limited by their scope—recomputing only selected tokens or adjusting attention distributions cannot fully restore the nuanced cross-chunk dependencies. This results in a performance ceiling around 5% improvement, insufficient for high-stakes applications. The fundamental problem is balancing efficiency with the preservation of semantic coherence across chunks.

The paper introduces a multi-strategy framework that combines recomputation and attention reshaping, leveraging their mutual strengths. Key innovations include:

• �� Dynamic token selection based on attention deviation metrics (ΔK), enabling targeted recomputation of critical tokens.
• �� Multi-layered attention reshaping to approximate full prefill attention patterns.
• �� Cross-model cache reuse via Droidspeak, which intelligently switches between reuse and recomputation across layers.

These strategies collectively aim to recover cross-chunk dependencies more effectively than prior isolated approaches, pushing the accuracy boundary beyond 5%.

• �� Models: Llama3.1-8B, Qwen-8B, Mistral-7B, evaluated on WikiMQA, Musique, RULER datasets.
• �� Metrics: F1 score, adjusted to exclude queries where baseline yields zero.
• �� Techniques:
• Recomputing tokens with high ΔK at specific layers, selecting 15% of tokens for targeted KV cache updates.
• Attention reshaping via APE and SEL to modify attention scores, mimicking full prefill patterns.
• Dynamic token selection based on layer-wise ΔK analysis, adjusting recomputation scope.
• Multi-model collaboration (Droidspeak) for cross-model cache reuse, switching between reuse and recomputation.
• �� Experiments involve ablation studies, hyperparameter tuning (p=1-10), and cross-scenario validation to measure accuracy improvements.

The experimental setup involves testing multiple models on datasets emphasizing multi-hop reasoning. The baseline is full prefill, with comparisons against naive reuse, Cacheblend, EPIC, Cacheclip, and the proposed combined approach. Hyperparameters such as recomputation ratio (15%) and token selection thresholds are tuned for optimal performance. Accuracy is measured via F1 scores, with particular attention to hard queries (F1=0). Results show that combined strategies outperform single methods, with an average accuracy gain of 5%. Ablation studies confirm the importance of dynamic token selection and multi-model collaboration. The robustness across datasets and models demonstrates the approach’s generalizability.

The integrated approach achieves up to 5% higher accuracy than individual techniques across models and datasets. For instance, in Llama on WikiMQA, accuracy improved from 0.66 to 0.72; on Qwen, from 0.66 to 0.78. Ablation results highlight that dynamic token selection and multi-model cache reuse are critical for performance gains. The results also reveal that addressing attention sink phenomena at chunk starts and dynamically adjusting recomputation scope are essential for recovering cross-chunk dependencies. Overall, the experiments validate that combining recomputation and reshaping strategies effectively overcomes the limitations of single techniques, especially in multi-hop reasoning tasks.

This approach is suited for retrieval-augmented generation (RAG) systems, question answering, summarization, and multi-modal reasoning tasks where efficiency and accuracy are paramount. It requires efficient cache management, dynamic token selection, and multi-model infrastructure. Industry applications include intelligent assistants, search engines, and content creation tools that demand fast, accurate responses in complex contexts. The methodology enables deployment in scenarios with limited computational resources by optimizing cache reuse without sacrificing semantic coherence.

Despite improvements, the approach still struggles with extremely long or highly complex texts where cross-chunk dependencies are inherently difficult to recover. Additional computational overhead from multi-strategy fusion may impact real-time performance. The experiments are primarily on specific models and datasets, so generalization to other architectures or modalities remains uncertain. Future work should focus on optimizing efficiency, reducing costs, and extending applicability to broader tasks and larger models.