EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
EVICPRESS jointly optimizes KV-cache compression and eviction, reducing inference latency while maintaining quality.
Key Findings
Methodology
EVICPRESS employs a unified utility function to quantify the impact of lossy compression and eviction on generation quality and delay. It periodically profiles all contexts, evaluating possible configurations across multiple storage tiers via a heuristic algorithm that maximizes overall utility. The system applies conservative compression for sensitive contexts and aggressive compression for less sensitive ones, enabling global optimization. The approach integrates with vLLM, intercepting cache operations and dynamically adjusting cache placement based on real-time profiling, ensuring a balanced trade-off between latency and quality. Extensive experiments across 12 datasets and 5 models demonstrate significant improvements in inference speed and cache hit rates.
Key Results
- On five models including LLaMA-7B and Qwen-3-32B, EVICPRESS achieves up to 2.19× faster time-to-first-token (TTFT) with less than 3% quality degradation. Across datasets like Samsum, TriviaQA, and MultiNews, the system improves cache hit rates by 15-25%, reduces average latency, and enhances throughput by 2.0–3.6× compared to static baselines. The adaptive configuration effectively balances sensitivity, maintaining high quality for critical contexts while aggressively compressing less sensitive ones.
- Ablation studies confirm that the joint optimization framework outperforms separate compression or eviction strategies, with a consistent TTFT gain of over 1.5×. The periodic re-profiling mechanism adapts to query distribution shifts, sustaining performance over time. Results indicate that the utility function accurately captures the trade-offs, guiding near-optimal cache management decisions.
- The experiments validate EVICPRESS’s robustness across diverse scenarios, showing its potential to significantly reduce hardware costs and improve user experience in large-scale AI services.
Significance
This work addresses a critical bottleneck in deploying large-scale language models—efficient cache management across hierarchical storage. By jointly optimizing compression and eviction, EVICPRESS offers a systematic solution to reduce inference latency and hardware costs without sacrificing quality. Its innovative use of a utility function for global decision-making marks a paradigm shift from traditional static policies, enabling adaptive, context-sensitive management. The approach is highly scalable and applicable to real-world AI systems, promising broader impacts in cloud AI deployment, edge computing, and multi-modal models. As models continue to grow, such intelligent cache management frameworks will be essential for sustainable AI infrastructure development.
Technical Contribution
The paper introduces a novel multi-objective utility function that quantifies the combined effects of lossy compression and eviction on inference quality and delay. It develops a dynamic profiling and re-configuration pipeline that periodically updates cache configurations based on real-time query distributions. The core algorithm employs a greedy heuristic to select configurations maximizing the utility across multiple storage tiers, considering context sensitivities. The implementation extends vLLM with multi-tier cache control, integrating compression algorithms like keydiff, quantization, and token dropping, with adaptive ratios. The system demonstrates theoretical guarantees of near-optimality under certain assumptions and achieves practical efficiency through heuristic search. This comprehensive framework surpasses prior static or isolated strategies, enabling holistic cache management in large-scale inference systems.
Novelty
This is the first work to formalize and implement a joint optimization framework for lossy compression and hierarchical eviction in multi-tier KV caches for LLM inference. Unlike previous approaches that treat compression and eviction separately, EVICPRESS models their interactions via a unified utility function, capturing the trade-offs between quality and latency. Its dynamic profiling and re-configuration mechanism adaptively responds to query distribution shifts, ensuring sustained performance. The integration of multi-objective optimization with heuristic search in this context represents a significant advancement, setting a new standard for cache management in AI inference systems.
Limitations
- The effectiveness depends on the accuracy of the utility function, which may require tuning for different models or tasks. In highly dynamic environments, frequent re-profiling could introduce overhead.
- The system's complexity increases with the number of contexts and configurations, potentially impacting scalability in extremely large deployments.
- Sensitive contexts still pose challenges; overly aggressive compression might occasionally degrade quality beyond acceptable thresholds, necessitating further refinement of sensitivity detection.
Future Work
Future directions include integrating machine learning models to predict context sensitivity and optimal configurations more accurately, reducing reliance on periodic profiling. Extending the framework to multi-modal models and heterogeneous hardware environments will broaden its applicability. Additionally, exploring reinforcement learning techniques to dynamically balance trade-offs could further enhance system adaptability and robustness, enabling real-time, self-optimizing cache management in evolving inference workloads.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized natural language processing, but deploying these models efficiently remains a significant challenge. Central to this challenge is managing the KV cache, which stores intermediate states to accelerate inference. As models scale up and user demand increases, the cache footprint often exceeds GPU memory, creating a bottleneck that hampers response times and increases hardware costs. Traditional solutions, such as simple eviction policies or uniform compression, fall short in balancing speed and quality, especially in multi-tier storage hierarchies.
This paper introduces EVICPRESS, a novel system that jointly optimizes KV cache compression and eviction across multiple storage tiers. By employing a unified utility function that quantifies the impact of each configuration on inference delay and generation quality, EVICPRESS dynamically selects the best cache management strategy. The system periodically profiles all contexts, evaluates configurations through heuristic algorithms, and reconfigures cache placement in real-time. This adaptive approach allows the system to handle context sensitivities and query distribution shifts effectively.
Experimental results across 12 datasets and 5 models demonstrate that EVICPRESS can achieve up to 2.19× faster TTFT while maintaining high quality. It significantly outperforms static or single-strategy baselines, reducing latency and increasing throughput. The approach’s scalability and robustness make it suitable for industrial deployment, promising substantial improvements in AI inference efficiency.
Looking ahead, future work will focus on enhancing sensitivity prediction, extending to multi-modal and heterogeneous hardware settings, and employing reinforcement learning for autonomous optimization. EVICPRESS marks a significant step toward intelligent, scalable cache management for large AI models, enabling faster, more cost-effective deployment of next-generation language systems.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity, so prior work has proposed to either evict KV cache to lower-tier storage devices, or compress KV cache so that more KV cache can be fit in the fast memory. However, prior work misses an important opportunity: jointly optimizing the eviction and compression decisions across all KV caches to minimize average generation latency without hurting quality. We propose EVICPRESS, a KV-cache management system that applies lossy compression and adaptive eviction to KV cache across multiple storage tiers. Specifically, for each KV cache of a context, EVICPRESS considers the effect of compression and eviction of the KV cache on the average generation quality and delay across all contexts as a whole. To achieve this, EVICPRESS proposes a unified utility function that quantifies the effect of quality and delay of the lossy compression or eviction. To this end, EVICPRESS's profiling module periodically updates the utility function scores on all possible eviction-compression configurations for all contexts and places KV caches using a fast heuristic to rearrange KV caches on all storage tiers, with the goal of maximizing the utility function scores on each storage tier. Compared to the baselines that evict KV cache or compress KV cache, EVICPRESS achieves higher KV-cache hit rates on fast devices, i.e., lower delay, while preserving high generation quality by applying conservative compression to contexts that are sensitive to compression errors. Evaluation on 12 datasets and 5 models demonstrates that EVICPRESS achieves up to 2.19x faster time-to-first-token (TTFT) at equivalent generation quality.