LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
LMCACHE employs batch data movement and modular APIs to enable efficient cross-engine and hierarchical KV cache management for enterprise-scale LLM inference.
Key Findings
Methodology
This work introduces a high-performance KV cache layer utilizing batch operations, compute-I/O pipelining, and zero-copy transfers. It features a modular connector interface compatible with evolving inference engines like vLLM and SGLang, supporting hierarchical storage (GPU, CPU, remote) and cross-engine transfer. The system orchestrates cache offloading, prefill, and transfer via a unified API, enabling scalable, flexible management. Extensive experiments demonstrate up to 15× throughput gains in multi-turn QA and document analysis, with significant latency reductions. The design emphasizes data batching, parallelism, and adaptive scheduling to optimize bandwidth utilization and system responsiveness.
Key Results
- In diverse workloads, LMCACHE achieves up to 15× throughput improvement, with transfer rates reaching 49GBps using large chunk batching, outperforming traditional methods (~1GBps). Latency drops by over 50%, especially in cross-node and remote storage scenarios. Cache hit ratios are maintained despite content truncation, thanks to efficient prefill and cache reuse strategies. Compatibility with multiple models and inference frameworks confirms system robustness and scalability.
- Real-world deployment shows remote storage prefetch reduces fill delay significantly, and content truncation improves cache efficiency. The system supports multi-user, long-context scenarios with high reuse rates, validating its effectiveness in enterprise environments. API flexibility enables custom cache management policies, enhancing integration and operational control.
- Compared to native inference APIs and open-source frameworks, LMCACHE offers superior throughput and latency, especially under high concurrency. Its modular architecture and multi-layer storage support future hardware and model evolution, setting a new standard for scalable, efficient KV caching in large-scale inference systems.
Significance
This research addresses critical bottlenecks in large model inference, where KV cache size and transfer efficiency limit scalability. By enabling hierarchical, cross-engine, and remote cache management, LMCACHE significantly enhances throughput, reduces latency, and supports long-context, multi-user applications. Its open-source design encourages industry adoption and standardization, fostering broader deployment of enterprise AI solutions. The system’s flexibility and performance improvements lay a foundation for future innovations in large-scale model serving, making it a pivotal contribution to AI infrastructure.
Technical Contribution
The paper introduces a comprehensive KV cache management framework combining batch processing, compute-I/O overlapping, and modular interfaces. It employs large chunk batching, zero-copy transfers, and asynchronous layer-wise pipelining to maximize bandwidth utilization. The modular connector decouples system from inference engine specifics, facilitating rapid adaptation. Hierarchical storage APIs enable seamless cross-node and remote cache transfer, supporting multi-model, multi-task scenarios. These innovations collectively push the boundary of scalable, flexible KV cache systems for enterprise inference.
Novelty
This work is the first to unify batch data movement, multi-layer storage, and modular API design into a scalable KV cache framework tailored for enterprise inference. Unlike prior solutions limited to single-node or static storage, LMCACHE supports dynamic, cross-engine, and remote cache management with high throughput. Its integration of large chunk batching and compute-I/O pipelining sets new standards in efficiency, enabling real-world deployment at scale.
Limitations
- While highly optimized, the system may face challenges under extremely high concurrency or in environments with limited network bandwidth, affecting transfer rates and consistency.
- Dependence on hierarchical storage introduces complexity; bandwidth bottlenecks at remote storage can limit scalability.
- Handling ultra-large models or extremely long contexts may incur high storage and transfer costs, requiring further compression and scheduling improvements.
Future Work
Future directions include enhancing multi-node consistency, adaptive scheduling, and intelligent cache prediction. Integrating hardware accelerators and exploring AI-driven cache management can further improve performance. Extending support for heterogeneous storage technologies and refining algorithms for ultra-large models will be key to pushing the system’s capabilities.
AI Executive Summary
The rapid growth of large language models (LLMs) has transformed AI applications across industries, but their inference systems face significant scalability challenges. Central to these challenges is the management of KV caches, which store intermediate attention states. Traditionally confined within GPU memory, KV caches are now expanding beyond capacity due to longer contexts and increased traffic, necessitating more efficient offloading and transfer mechanisms.
This paper introduces LMCACHE, a high-performance, modular KV cache layer designed for enterprise-scale inference. By leveraging batch data movement, compute-I/O pipelining, and zero-copy transfers, LMCACHE enables seamless cross-engine and hierarchical storage management. Its standardized connector interface supports rapid adaptation to evolving inference frameworks like vLLM and SGLang, ensuring compatibility and extensibility. The system orchestrates cache offloading, prefill, and transfer through a unified API, facilitating flexible, scalable deployment.
Extensive experiments demonstrate that LMCACHE, combined with vLLM, achieves up to 15× throughput improvements in multi-turn question answering and document analysis tasks, with latency reductions exceeding 50%. These results highlight the system’s ability to significantly boost inference efficiency, reduce operational costs, and enable long-context, multi-user applications in enterprise environments.
Beyond performance, LMCACHE’s design emphasizes adaptability and future-proofing. Its multi-layer storage support, cross-node transfer capabilities, and API flexibility position it as a foundational infrastructure for next-generation AI deployment. The open-source release encourages industry adoption, standardization, and further innovation.
Looking ahead, future work will focus on multi-node consistency, AI-driven cache prediction, and hardware acceleration integration. Addressing these areas will further enhance scalability, robustness, and efficiency, ensuring that large models can meet the demands of real-world enterprise applications at scale.
Deep Dive
Abstract
KV cache has traditionally been stored in GPU memory to accelerate the decoding phase of large language model (LLM) inference. However, it is increasingly necessary to move KV caches outside GPU devices, to enable cache reuse across different queries and inference engines. Our real-world usage statistics confirm this trend: over time, the total KV cache stored by users has grown rapidly, far exceeding the capacity of GPU memory. Despite this need, there lacks an efficient solution for offloading and transferring KV caches. We present LMCACHE, the first and so far the most efficient open-source KV caching solution, which extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries. LMCACHE supports both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation (cross-engine/GPU cache transfer). LMCACHE's high performance and wide adoption stem from the following contributions: (1) highly optimized KV cache data movement powered by batched data movement operations, compute and I/O pipelining; (2) a modular KV cache connector component, decoupling LMCACHE from the rapid evolution of inference engines; (3) a first-class control API for flexible cache orchestration across GPU, CPU, storage, and network layers. Our evaluation shows that combining LMCACHE with vLLM achieves up to 15x improvement in throughput across workloads such as multi-round question answering and document analysis. Large-scale adoption of LMCACHE in enterprise settings provides us valuable insights, for example, fetching KV cache from remote storage has unsurprisingly benefits to prefill delay, and that context truncation, which is a widely applied technique in industry, can greatly reduce prefix cache hit ratio by half. The source code of LMCACHE is at: https://github.com/LMCache/LMCache.