PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving

The evolution of large language models (LLMs) like GPT-3, LLaMA, and Qwen has significantly advanced NLP capabilities. However, their deployment in real-world applications is hindered by computational and storage bottlenecks, especially during inference. Techniques such as KV cache reuse, sparse attention, and model pruning have been proposed to improve efficiency, but long input sequences—common in retrieval-augmented generation (RAG)—still cause high latency. RAG systems combine external knowledge retrieval with LLMs, enabling more accurate responses without retraining, but the extended input length exacerbates prefill delays. Existing cache management strategies, limited by GPU memory and data transfer costs, cannot fully address these challenges, necessitating innovative multi-tier storage and asynchronous processing solutions.

The core issue lies in the prefill stage of RAG inference, where generating KV caches for long inputs incurs substantial latency. Reusing shared prefixes helps reduce this, but current methods are constrained by GPU memory limits and high data transfer overheads between CPU, GPU, and SSD. Synchronous data movement causes idle GPU time, reducing throughput. As input length grows, the latency increases super-linearly, severely impacting real-time performance. Efficiently managing large-scale KV caches across multiple storage layers while minimizing transfer costs remains an open challenge.

This paper introduces three main innovations: 1) a prefix-tree structure for fast KV cache matching, combined with a look-ahead LRU policy to improve cache hit ratios; 2) a layer-wise overlapping technique that pipelines KV cache loading, model computation, and cache offloading across CUDA streams, reducing communication overhead; 3) a queue-based prefetching mechanism that loads KV caches from SSD into DRAM during request wait times, hiding SSD latency. These components work together to extend cache capacity beyond GPU memory, optimize data movement, and significantly reduce prefill latency, enabling scalable, low-latency inference for large models.

• �� Organize long input sequences into fixed-size chunks and build a prefix tree for efficient matching. • Use a look-ahead LRU policy that leverages pending request information to prioritize cache retention for soon-to-be-reused chunks. • Implement layer-wise overlapping, executing cache loading, model inference, and cache offloading in parallel across CUDA streams to hide communication delays. • Design a queue-based prefetching thread that, during request waiting, preloads KV caches from SSD into DRAM based on predicted future requests. • Integrate these strategies into a unified system on top of vLLM, supporting multi-layer storage management and asynchronous operations, to maximize cache reuse and minimize latency.

Experiments used Llama2-7B and Qwen-14B models with real-world long input requests from RAG scenarios. Baselines included traditional KV cache reuse methods. Metrics focused on TTFT, throughput, and cache hit rate. Different storage configurations (GPU, DRAM, SSD) tested system scalability. Ablation studies isolated the impact of each component—prefix tree, layer-wise overlap, prefetching—demonstrating their individual and combined contributions. Hyperparameters tuned for optimal performance, with extensive stress testing under high request concurrency to evaluate robustness.

PCR achieved an average 15% TTFT reduction, with peak speedup of 2.47× compared to baseline systems. The prefix tree with look-ahead LRU improved cache hit rate by over 20%. Layer-wise overlapping reduced GPU idle time by 30%, while prefetching from SSD cut load latency by 40%. These results confirm that the integrated approach effectively minimizes prefill delays, enhances throughput, and scales well with large request volumes, validating its practical applicability in industrial settings.

The system is suitable for real-time NLP applications such as chatbots, knowledge retrieval, and content generation, where long inputs and external knowledge are essential. It can be integrated into cloud-based inference servers, supporting large-scale deployment with minimal latency. Future extensions could include multi-model support, adaptive prefetching algorithms, and broader storage hierarchies, enabling broader industrial adoption and more intelligent resource management.

The prefix-based matching may be less effective with highly diverse or unpredictable requests, reducing cache reuse efficiency. The prefetching relies on request prediction, which can misfire, leading to unnecessary data transfers. SSD bandwidth and latency still pose challenges under extreme load, and the system's complexity increases with storage hierarchy expansion. Future work should focus on adaptive algorithms and hardware innovations to further mitigate these issues.