ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL
ROSE uses cooperative elasticity on serving GPUs for Agentic RL, boosting throughput by 1.3-3.3x.
Key Findings
Methodology
ROSE system achieves cooperative elasticity for Agentic RL post-training, comprising three components: SLO-safe co-serving executor, cross-cluster weight transfer engine, elastic rollout scheduler. The executor dynamically shares memory and compute on the same GPUs, the transfer engine leverages shard-aware routing and weight sparsity for fast synchronization, and the scheduler dynamically routes rollouts.
Key Results
- Experiments across multiple model sizes and cluster scales show ROSE improves end-to-end throughput by 1.3-3.3x over resource-fixed baselines and reduces rollout time by 1.2-1.5x over resource-elastic baselines, with no serving SLO violations.
- ROSE significantly enhances training efficiency across various models and cluster scales, especially under fluctuating resource demands.
- The cross-cluster weight transfer engine enables rapid synchronization, minimizing communication overhead.
Significance
ROSE addresses the challenge of fluctuating resource demands in Agentic RL training by sharing idle serving GPU resources, significantly boosting training efficiency and reducing computational overhead. This method holds substantial significance in both academia and industry, particularly in post-training optimization of large language models.
Technical Contribution
ROSE introduces a novel cooperative elasticity framework, overcoming the limitations of existing fixed-resource methods and providing a solution for dynamic resource allocation. Through innovative cross-cluster weight transfer and co-serving mechanisms, ROSE achieves more efficient resource utilization.
Novelty
ROSE is the first to propose cooperative elasticity by sharing rollout workloads on serving GPUs, significantly reducing allocation overhead and enhancing resource utilization compared to existing elastic methods.
Limitations
- Under peak traffic, maintaining service SLOs can be challenging, despite ROSE's priority control mechanisms.
- Cross-cluster communication still has latency, potentially affecting real-time performance.
- Further optimization is needed to adapt to larger-scale models and clusters.
Future Work
Future work could explore applications on larger-scale clusters and further optimize cross-cluster communication mechanisms to reduce latency and enhance real-time performance.
AI Executive Summary
The ROSE system employs cooperative elasticity on serving GPUs for Agentic RL post-training, addressing the challenges posed by fluctuating resource demands. Traditional fixed-resource systems cannot adapt to these variations, while existing elastic methods suffer from high allocation overhead and limited availability. ROSE achieves dynamic resource allocation by sharing idle GPU resources from serving clusters, significantly enhancing training efficiency. Experimental results demonstrate that ROSE improves end-to-end throughput by 1.3-3.3x and reduces rollout time by 1.2-1.5x across various model sizes and cluster scales. Although ROSE faces challenges in maintaining service SLOs, its innovative co-serving mechanism and cross-cluster weight transfer engine provide effective solutions. Future research directions include applications on larger-scale clusters and further optimization of cross-cluster communication mechanisms.
Deep Analysis
Background
Agentic RL is transforming post-training processes for large language models, with training time dominated by compute-intensive multi-turn rollouts and significant resource demand variations across training steps. Fixed-resource systems cannot adapt to these variations, while elastic methods face high allocation overhead and limited availability.
Core Problem
The rollout stage in Agentic RL training accounts for over 70% of total time, with significant GPU compute and memory demand variations across training steps. Existing fixed-resource systems cannot adapt to these demand variations, while elastic systems face high allocation overhead.
Innovation
ROSE introduces cooperative elasticity by sharing idle serving GPU resources, proposing a novel dynamic resource allocation method. The co-serving executor dynamically shares memory and compute on the same GPUs, the cross-cluster weight transfer engine leverages shard-aware routing and weight sparsity for fast synchronization, and the elastic rollout scheduler dynamically routes rollouts.
Methodology
- �� SLO-safe co-serving executor: dynamically shares memory and compute resources, preserving service SLOs.
- �� Cross-cluster weight transfer engine: leverages shard-aware routing and weight sparsity for fast synchronization.
- �� Elastic rollout scheduler: dynamically routes rollouts to dedicated and opportunistic serving GPUs.
Experiments
Experiments conducted across various model sizes and cluster scales using Qwen3-8B and Qwen3-32B models validate ROSE's performance improvements under different resource configurations. Baselines include fixed-resource and elastic systems.
Results
ROSE improves end-to-end throughput by 1.3-3.3x and reduces rollout time by 1.2-1.5x in experiments. The cross-cluster weight transfer engine enables rapid synchronization, minimizing communication overhead.
Applications
ROSE is applicable for post-training optimization of large language models, especially under fluctuating resource demands. Its dynamic resource allocation method significantly enhances training efficiency and reduces computational overhead.
Limitations & Outlook
Although ROSE faces challenges in maintaining service SLOs, its innovative co-serving mechanism and cross-cluster weight transfer engine provide effective solutions. Future research directions include applications on larger-scale clusters and further optimization of cross-cluster communication mechanisms.
Plain Language Accessible to non-experts
Imagine a kitchen where chefs need to prepare different numbers of dishes at different times. Traditional methods allocate fixed stoves and utensils to each chef, leading to resource waste as sometimes chefs don't need all the equipment. ROSE is like a smart kitchen management system that dynamically allocates stoves and utensils based on the chefs' needs, ensuring each chef has enough resources when needed. This approach not only improves efficiency but also reduces resource waste.
ELI14 Explained like you're 14
Imagine playing a game where you need different weapons and gear for different levels. Traditional methods give you fixed gear every time, which can be insufficient or wasteful. ROSE is like a smart gear management system that dynamically allocates weapons and gear based on level needs, ensuring you have enough resources when needed. Isn't that cool? It not only improves game efficiency but also reduces resource waste.
Glossary
Agentic RL
A reinforcement learning method emphasizing proactive interaction between agents and environments.
Used in the paper to optimize post-training processes for large language models.
SLO
Service Level Objective, a target metric for service performance such as response time.
Used in the paper to measure serving cluster performance.
GPU
A hardware device used to accelerate computation, especially in deep learning.
Used in the paper to execute compute tasks for Agentic RL.
Rollout
The interaction process between agents and environments in reinforcement learning.
Used in the paper to describe the training stage of Agentic RL.
Elasticity
The ability to dynamically adjust resources based on demand.
Used in the paper to describe the core feature of the ROSE system.
Open Questions Unanswered questions from this research
- 1 How to further optimize cross-cluster communication mechanisms to reduce latency?
- 2 What are the challenges of applying ROSE to larger-scale clusters?
- 3 How to maintain service SLOs under peak traffic?
Applications
Immediate Applications
Large Language Model Optimization
ROSE can be used to optimize post-training processes for large language models, especially under fluctuating resource demands.
Long-term Vision
Intelligent Resource Management
ROSE's dynamic resource allocation method can be applied to broader fields of computational resource management, driving intelligent development.
Abstract
Agentic reinforcement learning (RL) is reshaping LLM post-training, but end-to-end training time is dominated by compute-intensive, multi-turn rollouts whose resource demand varies significantly across training steps. Resource-fixed systems cannot adapt to this variation, while resource-elastic approaches that provision external GPUs on demand suffer from high allocation overhead and limited availability. We observe that serving clusters leave substantial GPU compute and memory idle, and propose cooperative elasticity: sharing already-deployed serving GPUs with rollout workloads to provide on-demand elastic capacity. Realizing this is non-trivial, as it must preserve serving SLOs under bursty traffic while minimizing cross-cluster communication overhead. We present ROSE, a system that realizes cooperative elasticity for agentic RL post-training, comprising three components: (1) an SLO-safe co-serving executor that co-locates heterogeneous serving and rollout models on the same GPUs, dynamically sharing memory and compute while preserving serving SLOs; (2) a cross-cluster weight transfer engine that leverages shard-aware routing and weight sparsity for fast synchronization; and (3) an elastic rollout scheduler that dynamically routes rollouts across dedicated and opportunistic serving GPUs. Experiments across multiple model sizes and cluster scales show that ROSE improves end-to-end throughput by 1.3 - 3.3 x over resource-fixed baselines and reduces rollout time by 1.2 - 1.5 x over resource-elastic baselines, with no serving SLO violations.