TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
TokenFlow employs preemptive scheduling and proactive KV cache management to boost streaming throughput by 82.5%, reducing P99 TTFT by 80.2%.
Key Findings
Methodology
TokenFlow integrates a buffer-aware preemptive scheduler with proactive hierarchical KV cache management. The scheduler dynamically prioritizes requests based on real-time token buffer occupancy and consumption rates, enabling request preemption and resumption. Background asynchronous KV cache transfer overlaps I/O with computation, minimizing overhead. The system architecture includes request tracking, dynamic scheduling, request offload management, LLM inference, and hierarchical cache control, ensuring high resource utilization and low latency. The scheduling algorithm adjusts priorities to match token generation and consumption, while cache management employs chunked asynchronous transfers to reduce I/O costs. This tight integration allows for responsive handling of request bursts, balancing throughput and latency effectively.
Key Results
- Experiments on Llama3-8B and Qwen2.5-32B models across RTX 4090, A6000, and H200 GPUs show up to 82.5% increase in effective throughput and 80.2% reduction in P99 TTFT. The system maintains high overall throughput while significantly improving user-centric metrics. Under burst scenarios, request queuing delays are minimized, and resource utilization is optimized. Background cache transfers reduce migration overhead by over 20%, enabling smoother request handling. The results demonstrate the system’s robustness across diverse workloads and hardware configurations.
Significance
This work addresses fundamental challenges in real-time LLM text streaming, notably balancing low latency and high throughput during request bursts. By introducing a buffer-aware preemptive scheduling framework coupled with proactive cache management, it overcomes limitations of traditional non-preemptive and reactive systems. The approach enhances resource efficiency, reduces user-perceived latency, and improves overall system responsiveness, making large-scale, real-time AI services more feasible. It paves the way for deploying LLMs in latency-sensitive applications like conversational agents, live translation, and voice assistants, with broad industry implications.
Technical Contribution
The core technical innovation lies in the design of a dynamic, buffer-aware preemptive scheduler that adjusts request priorities based on real-time buffer states and consumption rates. This scheduler is tightly integrated with a hierarchical KV cache management system that asynchronously transfers key-value pairs between GPU and CPU memory, reducing migration overhead. The system’s architecture enables overlapping I/O with inference computation, effectively hiding memory transfer latency. The combination of these components provides a novel end-to-end solution for balancing responsiveness and throughput in bursty workloads, with formal guarantees on resource utilization and latency bounds. This represents a significant advancement over existing static or reactive scheduling and caching strategies.
Novelty
This is the first work to combine buffer-aware preemptive scheduling with proactive hierarchical KV cache management specifically for LLM text streaming under bursty workloads. Unlike prior approaches that focus solely on throughput or latency, TokenFlow emphasizes user experience by dynamically balancing both. Its innovative scheduling algorithm considers real-time buffer states and consumption rates, enabling request preemption without sacrificing overall throughput. The integration of background cache migration and overlapping I/O with inference constitutes a new engineering paradigm, filling a research gap in efficient, user-centric LLM serving.
Limitations
- Despite its improvements, TokenFlow may still face GPU memory constraints under extremely high request volumes, where cache migration overhead could become significant. The scheduling parameters require careful tuning for different workloads, limiting out-of-the-box deployment. Additionally, the current design mainly targets single-node setups; multi-node scalability and consistency are future challenges.
Future Work
Future research will explore multi-GPU and distributed deployment strategies to enhance scalability. Developing predictive models for request load and cache needs can further optimize scheduling decisions. Hardware acceleration techniques, such as NVMe-based caching or specialized inference chips, could reduce migration overhead. Extending the framework to multi-node environments and integrating with cloud orchestration tools will be key steps toward industrial-scale deployment.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized natural language processing, enabling applications like chatbots, real-time translation, and voice assistants. However, deploying these models in real-time interactive settings presents significant challenges. Traditional serving systems often struggle under request bursts, suffering from high latency and low resource utilization due to rigid scheduling and reactive memory management. These limitations hinder user experience and scalability.
TokenFlow introduces a novel architecture that combines preemptive request scheduling with proactive hierarchical KV cache management. The core idea is to dynamically prioritize requests based on real-time buffer occupancy and consumption rates, enabling the system to preempt and resume requests intelligently. Background asynchronous KV cache transfers between GPU and CPU further reduce migration overhead, overlapping I/O with inference computation. This integrated approach allows TokenFlow to adapt swiftly to workload fluctuations, maintaining low latency and high throughput.
Extensive experiments on multiple GPU platforms with models like Llama3-8B and Qwen2.5-32B demonstrate that TokenFlow achieves up to 82.5% higher effective throughput and reduces P99 TTFT by over 80%. These results highlight its ability to handle bursty workloads efficiently while ensuring a smooth user experience. The system’s design addresses core bottlenecks in current LLM serving architectures, offering a scalable, resource-efficient solution for real-time AI applications.
Looking ahead, future work will focus on extending multi-node scalability, integrating predictive load models, and leveraging hardware accelerators to further reduce latency and migration costs. Despite its promising performance, challenges remain in optimizing for extreme workloads and ensuring system robustness across diverse deployment environments. Nonetheless, TokenFlow marks a significant step toward responsive, efficient, and user-centric LLM serving at scale.
Deep Analysis
Background
随着深度学习模型不断扩大规模，LLM在自然语言处理中的应用逐渐普及。从GPT、LLaMA到Qwen等，模型性能不断突破，但在实际部署中，面临调度与内存管理的瓶颈。传统系统多采用非预占调度，导致请求排队延长，资源利用率低，用户体验下降。近年来，调度优化和异步缓存迁移成为研究热点，旨在提升系统响应速度和吞吐能力，但在突发请求场景下仍存在不足。现有方法难以兼顾低延迟与高吞吐，限制了大规模实时应用的推广。
Core Problem
在请求突发时，传统LLM服务系统表现出明显的响应延迟和资源浪费。非预占调度难以灵活应对高峰，导致排队时间长，用户体验差。预占调度虽能缓解排队，但频繁切换影响模型推理效率，GPU内存易因KV缓存增长超载。如何在保证低延迟的同时最大化资源利用，成为核心难题。现有系统缺乏动态调度与缓存优化的有效结合，限制了其应对突发负载的能力。
Innovation
本研究提出缓冲区感知预占调度算法，依据请求缓冲状态和生成速率动态调整优先级，结合主动KV缓存迁移机制，显著提升响应速度和吞吐。调度器实时监控请求缓冲区，提前预占资源，减少排队等待。缓存管理采用异步写入与加载策略，降低迁移开销。系统架构实现调度与缓存的深度融合，通过背景异步I/O隐藏迁移延迟，整体优化请求响应与内存利用。这一创新突破了传统被动调度和缓存策略的限制，提供端到端的高效解决方案。
Methodology
- �� 请求追踪器监控每个请求的状态，包括缓冲区、延迟目标、消费速率。
- �� 缓冲调度器根据实时指标动态调整请求优先级，预占高缓冲请求。
- �� 请求卸载管理器执行内存迁移，异步写入/加载KV缓存，减少迁移延迟。
- �� LLM执行器基于SGLang实现推理，生成流式tokens。
- �� 层级KV缓存管理器协调GPU与CPU间的KV迁移，优化内存利用。
- �� 系统通过背景异步I/O隐藏迁移开销，确保请求连续响应。
Experiments
采用Llama3-8B和Qwen2.5-32B模型，在RTX 4090、A6000和H200平台进行测试。对比基线系统，评估指标包括有效吞吐、TTFT和请求延迟。请求场景涵盖突发请求、连续负载和压力测试。参数调优涉及调度优先级阈值和缓存迁移策略，进行多轮消融验证各组件贡献。实验中，TokenFlow在多GPU环境下实现最高82.5%的有效吞吐提升，P99 TTFT降低80.2%，显著优于传统方案。背景缓存迁移策略降低迁移开销20%以上，整体资源利用率提升，用户体验明显改善。
Results
TokenFlow在多个GPU平台上实现了显著性能提升，最高82.5%的有效吞吐增长，P99 TTFT降低80.2%。在突发请求场景中，系统能快速响应，减少排队等待时间，资源利用率提升。背景异步缓存迁移降低迁移开销20%以上，整体系统效率增强。多场景验证显示其鲁棒性和适应性强，能在不同模型和硬件配置下保持优异表现，为工业应用提供可靠保障。
Applications
该系统适用于实时聊天机器人、语音助手、在线翻译等场景，能在高并发环境中保持低延迟和高吞吐。对硬件资源要求较低，兼容多GPU平台，适合大规模工业部署。未来可结合云端调度框架，支持多节点协同，推动行业智能化升级。系统还可扩展到多模态交互和多任务场景，满足未来智能服务的多样需求。
Limitations & Outlook
在极端请求高峰时，GPU内存压力仍可能成为瓶颈，缓存迁移频繁带来额外开销。调度参数需根据场景调优，系统在多节点环境中的扩展性和一致性仍待验证。未来需优化多GPU协作策略，降低迁移成本，提升系统弹性。
Abstract
Real-time LLM interactions demand streamed token generations, where text tokens are progressively generated and delivered to users while balancing two objectives: responsiveness (i.e., low time-to-first-token) and steady generation (i.e.,required time-between-tokens). Standard LLM serving systems suffer from the inflexibility caused by non-preemptive request scheduling and reactive memory management, leading to poor resource utilization and low request processing parallelism under request bursts. Therefore, we present TokenFlow, a novel LLM serving system with enhanced text streaming performance via preemptive request scheduling and proactive key-value (KV) cache management. TokenFlow dynamically prioritizes requests based on real-time token buffer occupancy and token consumption rate, while actively transferring KV cache between GPU and CPU memory in the background and overlapping I/O with computation to minimize request preemption overhead. Extensive experiments on Llama3-8B and Qwen2.5-32B across multiple GPUs (RTX 4090, A6000, H200) demonstrate that TokenFlow achieves up to 82.5% higher effective throughput (accounting for actual user consumption) while reducing P99 TTFT by up to 80.2%, without degrading overall token throughput.