ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
ThunderAgent employs program abstraction and dynamic scheduling to boost throughput by 1.5-3.6×, reducing KV cache thrashing and resource waste.
Key Findings
Methodology
The paper introduces the concept of modeling agent workflows as persistent LLM programs, integrating a program-aware scheduler and tool resource manager. The scheduler employs a cost-based optimization framework, utilizing a program abstraction that tracks metadata such as context length, execution phase, and resource dependencies. It dynamically pauses, resumes, and migrates programs across GPU nodes based on memory pressure and workload state, using mechanisms like periodic capacity checks, shortest-first eviction, and lifecycle-aware resource cleanup. Experiments across coding, routing, and scientific discovery tasks demonstrate the system's ability to improve throughput (1.5-3.6×), RL rollout efficiency (1.8-3.9×), and memory savings (up to 4.2×).
Key Results
- On SWE-Bench Lite, ThunderAgent achieved a 1.52× throughput increase, KV cache hit rate exceeded 85%, significantly reducing cache thrashing and re-prefill costs.
- In multi-node RL tasks, throughput was boosted to 3.9× compared to baseline systems, effectively alleviating cross-node memory imbalance.
- Asynchronous environment initialization reduced setup time by 30%, improving overall response speed and stability.
Significance
This work addresses fundamental limitations of request-level scheduling in large-scale agent inference, providing an end-to-end, program-centric framework that optimizes resource utilization, reduces cache thrashing, and manages tool environments efficiently. It paves the way for scalable, autonomous AI systems capable of handling complex multi-turn workflows with high throughput and stability, thus impacting both academia and industry by enabling more efficient deployment of large models in real-world applications.
Technical Contribution
The core innovation lies in abstracting agent workflows as persistent programs with metadata, enabling a unified scheduling approach. The system introduces a cost model guiding dynamic pausing, migration, and resource cleanup, supported by mechanisms like periodic capacity checks, shortest-first eviction, and lifecycle-aware garbage collection. These enable significant improvements over request-level systems, providing theoretical guarantees on cache efficiency and resource balancing, and supporting asynchronous environment setup.
Novelty
This is the first work to formalize agent workflows as persistent, schedulable programs, integrating program-aware scheduling with cross-node migration and lifecycle management. Unlike prior systems that treat requests independently, ThunderAgent captures workflow semantics, enabling end-to-end optimization that effectively mitigates cache thrashing and resource imbalance, representing a fundamental shift in large-scale LLM inference architecture.
Limitations
- Despite improvements, the system can still face scheduling delays under extremely high concurrency or highly heterogeneous workloads, requiring further adaptive strategies.
- Dependence on multi-GPU clusters increases deployment costs and complexity, limiting immediate applicability in resource-constrained environments.
- Long-duration tool environments may still introduce latency; future work should focus on more intelligent environment lifecycle management.
Future Work
Future directions include integrating reinforcement learning for adaptive scheduling policies, extending support for multi-modal workflows, and optimizing environment provisioning to further reduce latency and resource overhead. Additionally, exploring more robust mechanisms for handling extreme workloads and heterogeneous hardware will enhance system scalability and resilience.
AI Executive Summary
The rapid advancement of large language models (LLMs) has enabled complex multi-turn reasoning and autonomous agent workflows. However, existing inference systems, built from loosely coupled components like vLLM and Kubernetes, struggle to meet the demands of high throughput and resource efficiency. These systems treat each request independently, leading to cache thrashing, memory imbalance, and inefficient environment management, which significantly limit scalability.
To address these challenges, this paper introduces ThunderAgent, a novel system that models agent workflows as persistent, program-like entities. This abstraction allows for a unified view of heterogeneous resources such as KV caches, system states, and external tool environments. The core innovation is a program-aware scheduler that dynamically manages program states—pausing, migrating, and resuming programs based on real-time memory pressure and workload characteristics. This scheduler employs a cost model that balances decoding throughput against recomputation and resource wastage, optimizing overall system performance.
ThunderAgent also features a lifecycle-aware tool resource manager, which asynchronously initializes and recycles tool environments, preventing resource leakage and reducing setup times. The system's design supports cross-node migration, alleviating memory imbalance across GPU clusters, and employs a shortest-first eviction policy to minimize cache recomputation costs. Extensive experiments across coding, routing, and scientific discovery workloads demonstrate throughput improvements of up to 3.6×, RL rollout efficiency gains of 3.9×, and storage savings up to 4.2× compared to state-of-the-art baselines.
These results highlight ThunderAgent’s potential to significantly enhance the scalability and efficiency of large-scale agent inference systems. By integrating program semantics into scheduling and resource management, it opens new avenues for deploying autonomous AI at industrial scale. Despite some limitations in handling extreme workloads and environment complexity, the system establishes a robust foundation for future research in scalable, resource-aware AI inference architectures.
Deep Analysis
Background
近年来，随着GPT-4、PaLM等大规模语言模型的崛起，AI推理逐渐从单纯的问答转向多轮复杂交互。早期系统如Autellix和Continuum尝试通过GPU程序化和缓存机制优化多轮推理，但仍存在缓存穿透、内存不平衡和环境管理不足的问题。传统方法多依赖请求级调度，缺乏对工作流整体状态的感知，导致资源利用率低、扩展性差。随着模型规模和任务复杂度增加，亟需一种端到端的调度框架，既能优化缓存和资源，又能支持异步环境准备，满足大规模应用需求。
Core Problem
现有代理推理系统在多轮任务中面临三大瓶颈：KV缓存穿透导致频繁重填，降低吞吐；节点间内存不平衡引发资源浪费，影响系统扩展；工具环境生命周期管理不善，造成资源泄漏和环境准备延迟。这些问题在高并发和大规模部署中尤为突出，严重制约系统性能。解决方案需从工作流抽象、调度策略和资源管理三方面入手，设计端到端、程序感知的架构，以实现高效、稳定的推理服务。
Innovation
本研究提出将代理工作流抽象为持久化的LLM程序，作为调度单元，区别于传统请求级调度。引入程序感知调度器，结合成本模型，动态调节程序状态，减少KV穿透和节点内存不平衡，支持跨节点迁移。工具环境方面，设计生命周期感知的环境回收策略，异步初始化工具环境，避免资源泄漏。系统还实现了全局等待队列和短队列驱逐机制，有效优化资源利用和调度效率。这些创新显著提升了吞吐率和资源利用率，为大规模多轮推理提供了新思路。
Methodology
- �� 将代理工作流定义为持久化程序，包含唯一ID、上下文长度、工具环境、节点位置、执行状态。
- �� 构建成本模型，衡量解码、预填充、重计算、空闲缓存和资源利用的代价，指导调度优化。
- �� 设计程序感知调度策略，通过周期性检测KV缓存使用情况，动态暂停或迁移程序，避免缓存穿透。
- �� 实现跨节点迁移机制，将程序在不同GPU节点间迁移，缓解内存不平衡。
- �� 引入生命周期感知的工具资源管理，异步初始化和环境回收，减少资源泄漏。
- �� 采用最短队列优先策略，优先暂停短上下文程序，减少重填成本。
- �� 结合时间衰减机制，动态调整长时间空闲程序的优先级，优化内存利用。
Experiments
在SWE-Bench Lite、SWE-Agent、OpenHands和ScienceAgentBench等多个任务场景中，比较ThunderAgent与vLLM、Continuum等基线系统的性能。指标包括吞吐率、KV命中率、环境准备时间和存储占用。采用8×H100 GPU集群，调优参数如调度周期、队列阈值。还在大规模GPU集群上测试系统的扩展性，验证其在多节点环境中的性能表现。通过消融实验分析调度策略对性能的影响，确保系统在不同工作负载下均表现优越。
Results
ThunderAgent在多场景中实现了1.5-3.6倍的吞吐率提升，KV缓存命中率超过85%，显著减少缓存穿透。RL任务中，系统吞吐提升至原系统的3.9倍，有效缓解了跨节点内存不平衡问题。环境准备时间平均缩短30%，系统资源利用率提升20%以上。存储占用最高节省4.2倍，系统稳定性和扩展性得到验证，展现出优异的性能和应用潜力。
Applications
该系统适用于大规模多轮推理、强化学习、科学模拟等场景，特别适合需要高吞吐和资源优化的工业应用。通过端到端调度和资源管理，可显著降低硬件成本，提高系统响应速度，满足复杂任务的实时性需求。未来还可扩展到多模态、多任务环境，推动智能系统的自主化和规模化部署。
Limitations & Outlook
系统在极端高并发场景下仍存在调度延迟，调优复杂且依赖硬件环境。长时间工具环境可能引入延迟，调度策略参数需进一步优化。资源管理机制在极端情况下可能出现资源泄漏或环境初始化失败，未来需增强鲁棒性和自适应能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有很多不同的厨具和食材，每次做菜都需要准备食材、调料，还要用不同的厨具。有时候，某些厨具用完后还没清理干净，就会影响下一道菜的准备。传统做菜的方法是每次都重新准备所有东西，效率很低，也容易浪费材料。ThunderAgent就像一个聪明的厨师，它能记住每个厨具和食材的状态，知道什么时候需要清理，什么时候可以继续用。它还能根据菜的不同阶段，合理安排厨具和食材的使用，避免浪费和等待。这样一来，不仅做菜速度快了，材料也用得更省，厨房的资源得到了最优利用。这个系统让厨房变得井井有条，菜也做得更快更好。
ELI14 Explained like you're 14
想象你在学校的食堂帮忙准备饭菜，每次都要拿出新材料、洗菜、切菜，然后用锅炒。这很麻烦，而且浪费时间。有时候，你会发现某些材料还剩很多，但你不知道什么时候用完，结果就浪费了很多食材。ThunderAgent就像一个聪明的厨房助手，它记住每种材料的用量和什么时候用，能提前准备好食材，还会根据菜的不同阶段安排厨具。它还能在不同的厨房间调度厨具和食材，确保每个厨师都能顺利做菜，不会因为等待或材料短缺而耽误时间。这样一来，厨房的工作效率大大提高，菜也能更快做好，大家都很满意。它让厨房变得井井有条，节省了时间和材料，大家都觉得很神奇！
Abstract
Large language models(LLMs) are now used to power complex multi-turn agentic workflows. Existing systems run agentic inference by loosely assembling isolated components: an LLM inference engine (e.g., vLLM) and a tool orchestrator (e.g., Kubernetes). Although agentic workflows involve multiple LLM and tool requests, these systems schedule and allocate resources separately on a per-request basis, without end-to-end knowledge of the workflow. This leads to sub-optimal management of KV cache and tool execution environments. To address the challenges, we propose ThunderAgent, a fast, simple, and program-aware agentic inference system. We first abstract agentic workflows as LLM Programs, enabling a unified view of heterogeneous resources, including KV caches, system states, and external tool assets such as disk memory and network ports. Built upon this abstraction, ThunderAgent introduces a program-aware scheduler and a tool resource manager designed to maximize KV cache hit rates, mitigate memory imbalances, and enable asynchronous environment preparation. Evaluations across coding, routing, and scientific discovery agents demonstrate that ThunderAgent achieves 1.5-3.6x throughput improvements in serving, 1.8-3.9x in RL rollout, and up to 4.2x disk memory savings compared to state-of-the-art inference systems. To facilitate reproducibility and support future development, we open-source the system implementations of the whole ThunderAgent at: https://github.com/Agentic-Kinetics/ThunderAgent.