Stateful Large Language Model Serving with Pensieve
Pensieve employs multi-tier GPU-CPU caching and a generalized PagedAttention kernel to enable stateful LLM serving, achieving 1.14-3.0× throughput improvements in multi-turn conversations.
Key Findings
Methodology
This paper introduces Pensieve, a system that maintains conversation state across requests by leveraging a multi-tier GPU-CPU cache architecture. It caches previously processed token embeddings, reducing redundant computation during multi-turn dialogues. The system incorporates a novel generalized PagedAttention kernel capable of efficiently computing attention over non-contiguous memory regions, which is essential for handling partial cache evictions and scattered cached data. Additionally, Pensieve employs an integrated request batching and cache management strategy, including ahead-of-time swapping and pipelined data transfer, to optimize throughput and latency. The system manages cache eviction based on conversation activity and recomputation costs, ensuring efficient utilization of limited GPU memory. Extensive experiments on models like OPT-13B, Llama 2-13B, OPT-66B, and Llama 2-70B demonstrate significant performance gains over state-of-the-art stateless systems such as vLLM and TensorRT-LLM, with throughput improvements ranging from 1.14× to 3.0×.
Key Results
- In single-GPU models (OPT-13B, Llama 2-13B), Pensieve achieved throughput increases of 1.14-1.70×, while in multi-GPU models (OPT-66B, Llama 2-70B), the gains reached 1.64-3.0×, with notable latency reductions under moderate load.
- The cache reuse mechanism significantly decreased redundant context processing, with cache hit rates improving by over 20% and recomputation ratios dropping by 30%.
- The generalized non-contiguous attention kernel enabled efficient attention computation over scattered memory regions, contributing to overall system performance and scalability.
Significance
This work addresses the critical bottleneck in multi-turn dialogue systems—redundant processing of conversation history—by introducing a stateful serving architecture. It enables large language models to operate efficiently in real-time applications, making deployment of massive models feasible in resource-constrained environments. The approach bridges the gap between theoretical model capabilities and practical deployment, fostering advancements in AI-powered conversational agents, virtual assistants, and customer service bots. The innovations also open pathways for further research into cache-aware model serving, multi-modal integration, and adaptive resource management, with broad implications for both academia and industry.
Technical Contribution
The paper's main technical contributions include: 1) a multi-tier GPU-CPU cache system that stores and manages conversation history embeddings, reducing redundant computation; 2) a novel generalized PagedAttention kernel capable of handling attention over non-contiguous memory regions with multiple input tokens; 3) a unified request batching and cache management framework that supports both prefill and generation phases simultaneously; 4) an efficient cache eviction and recomputation strategy based on conversation activity and token importance. These innovations collectively enable high throughput and low latency in multi-turn dialogue scenarios, surpassing existing stateless serving systems.
Novelty
This work is pioneering in integrating multi-tier cache management with a generalized attention kernel to support stateful serving of large language models in multi-turn conversations. Unlike prior systems that treat each request independently, Pensieve maintains persistent context, significantly reducing redundant processing. The development of a non-contiguous memory-aware attention kernel further distinguishes this approach, addressing a key limitation in existing GPU kernels. The unified batching strategy across different conversation phases enhances hardware utilization, making it a novel contribution to the field of efficient LLM serving.
Limitations
- The system's performance may degrade with extremely long conversations exceeding cache capacity, necessitating frequent cache eviction and recomputation, which could impact latency.
- The complexity of cache management and scheduling introduces additional engineering challenges, requiring fine-tuning for different hardware configurations and workload patterns.
- Scaling to even larger models or multi-modal inputs may require further modifications to cache strategies and kernel implementations, limiting immediate applicability in diverse scenarios.
Future Work
Future research could explore adaptive cache management algorithms leveraging machine learning to predict conversation activity and optimize cache retention. Integrating model compression techniques such as pruning and quantization could further reduce memory footprint. Extending the system to support multi-modal inputs (images, audio) and multi-task learning would broaden its application scope. Additionally, deploying Pensieve in distributed multi-GPU clusters and evaluating its scalability and robustness in real-world scenarios remain important directions.
AI Executive Summary
The rapid growth of large language models (LLMs) like GPT-3, OPT, and Llama has revolutionized natural language processing, enabling applications such as chatbots, content generation, and code assistance. However, deploying these models in real-time, multi-turn conversational settings presents significant challenges. Traditional inference systems are stateless, requiring reprocessing the entire conversation history with each request, leading to redundant computation, increased latency, and limited scalability.
This paper introduces Pensieve, a novel system designed to address these bottlenecks by maintaining conversation state across requests through a multi-tier GPU-CPU cache architecture. By caching and reusing previously processed token embeddings, Pensieve reduces the need for repeated computation, significantly improving throughput and response times. The system incorporates a generalized PagedAttention kernel capable of efficiently computing attention over scattered, non-contiguous memory regions, which is crucial for handling partial cache evictions and scattered cached data.
A key innovation of Pensieve is its unified request batching strategy, which combines prefill and generation phases across multiple requests, maximizing GPU utilization and minimizing idle time. Cache management is further optimized through an eviction policy based on conversation activity and recomputation costs, ensuring efficient use of limited GPU memory. The system also employs ahead-of-time swapping and pipelined data transfer techniques to overlap computation and data movement, reducing latency.
Extensive experiments on models such as OPT-13B, Llama 2-13B, OPT-66B, and Llama 2-70B demonstrate that Pensieve achieves 1.14 to 3.0 times the throughput of state-of-the-art stateless systems like vLLM and TensorRT-LLM. Additionally, it significantly reduces response latency, especially under moderate load conditions. These results confirm that Pensieve effectively addresses the core inefficiency in multi-turn dialogue serving—redundant history processing—making large models more practical for real-world deployment.
Overall, Pensieve's combination of multi-tier caching, advanced attention kernels, and unified scheduling offers a scalable, efficient solution for stateful LLM serving. Its innovations pave the way for more responsive, resource-efficient AI assistants and conversational agents, with broad implications for industry and academia. Future work will focus on extending cache strategies, supporting multi-modal inputs, and scaling to larger models and distributed systems, further enhancing the capabilities of AI-driven dialogue systems.
Deep Dive
Abstract
Large Language Models (LLMs) are wildly popular today and it is important to serve them efficiently. Existing LLM serving systems are stateless across requests. Consequently, when LLMs are used in the common setting of multi-turn conversations, a growing log of the conversation history must be processed alongside any request by the serving system at each turn, resulting in repeated processing. In this paper, we design $Pensieve$, a system optimized for multi-turn conversation LLM serving. $Pensieve$ maintains the conversation state across requests by caching previously processed history to avoid duplicate processing. $Pensieve$'s multi-tier caching strategy can utilize both GPU and CPU memory to efficiently store and retrieve cached data. $Pensieve$ also generalizes the recent PagedAttention kernel to support attention between multiple input tokens with a GPU cache spread over non-contiguous memory. Our evaluation shows that $Pensieve$ can achieve $1.14$-$3.0\times$ the throughput of vLLM and TensorRT-LLM and significantly reduce latency.
References (20)
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, S. Ermon et al.
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang et al.
Clipper: A Low-Latency Online Prediction Serving System
D. Crankshaw, Xin Wang, Giulio Zhou et al.
Mixed Precision Training
P. Micikevicius, Sharan Narang, Jonah Alben et al.
DeepUM: Tensor Migration and Prefetching in Unified Memory
Jaehoon Jung, Jinpyo Kim, Jaejin Lee
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
Tianqi Chen, T. Moreau, Ziheng Jiang et al.
Low latency RNN inference with cellular batching
Pin Gao, Lingfan Yu, Yongwei Wu et al.
PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications
Zhihao Bai, Zhen Zhang, Yibo Zhu et al.
CacheGen: Fast Context Loading for Language Model Applications
Yuhan Liu, Han-Chiang Li, Kuntai Du et al.
LightSeq: A High Performance Inference Library for Transformers
Xiaohui Wang, Ying Xiong, Yang Wei et al.
Memory Harvesting in Multi-GPU Systems with Hierarchical Unified Virtual Memory
Sang-Jun Choi, Taeksoo Kim, Jinwoo Jeong et al.
INFaaS: Automated Model-less Inference Serving
Francisco Romero, Qian Li, N. Yadwadkar et al.
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
Paras Jain, Ajay Jain, Aniruddha Nrusimha et al.
Nexus: a GPU cluster engine for accelerating DNN-based video analysis
Haichen Shen, Lequn Chen, Yuchen Jin et al.
SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping
Chien-chin Huang, Guozhang Jin, Jinyang Li
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder et al.
InferLine: latency-aware provisioning and scaling for prediction serving pipelines
D. Crankshaw, Gur-Eyal Sela, Xiangxi Mo et al.
ZeRO-Offload: Democratizing Billion-Scale Model Training
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi et al.
Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture
S. Min, Kun Wu, Sitao Huang et al.
Cited By (20)
Pie: A Programmable Serving System for Emerging LLM Applications
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
A Generalized Memory Evaluation Method for Local Language Models with Applications in Critical Infrastructure Security
Network and Systems Performance Characterization of MCP-Enabled LLM Agents
LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market
TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
You Need an Encoder for Native Position-Independent Caching
Efficient Remote KV Cache Reuse with GPU-native Video Codec
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
ELORA: Efficient LoRA and KV Cache Management for Multi-LoRA LLM Serving
vEcho: A Paradigm Shift from Vulnerability Verification to Proactive Discovery with Large Language Models
PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems
Bat: Efficient Generative Recommender Serving with Bipartite Attention
Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
High Throughput and Low Latency LLM Serving via Adaptive KV Caching
TelePod: Live Migration for Stateful Containers