PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
PlexRL employs cluster-level scheduling to multiplex large language models across RLVR tasks, reducing GPU cost by 37.58%.
Key Findings
Methodology
This paper introduces PlexRL, a system that decouples RL algorithm control from model execution by abstracting model calls into a unified remote service. The core components include a cluster-wide scheduler, a remote LLM execution service with a stateless router and GPU-resident workers, and a centralized state manager. The scheduler optimizes model placement based on affinity and state-awareness, enabling dynamic multiplexing of large models across multiple RLVR tasks. The system supports model residency management, request scheduling, and state transitions, significantly reducing model migration overhead. Extensive experiments on RL tasks such as reasoning and agentic workflows demonstrate that PlexRL improves effective cluster capacity by up to 37.58%, enhances GPU utilization, and reduces GPU-hour costs, all while maintaining algorithmic flexibility.
Key Results
- In multiple RLVR workloads, PlexRL achieved a maximum of 37.58% reduction in GPU hours, with GPU utilization increasing by over 15%. Model migration frequency decreased by 30%, and resource fragmentation was significantly mitigated. The system demonstrated robustness across different model sizes, including GPT-3 and LLaMA, and various RL algorithms.
- The scheduling strategy enabled dynamic placement, effectively balancing workload demands and minimizing context switches. Results showed that resource reuse across tasks led to substantial efficiency gains, especially for large models with high state costs.
- Ablation studies confirmed that affinity-aware and state-sensitive scheduling were key to performance, with the system adapting well to workload heterogeneity and scale variations.
Significance
This work addresses fundamental inefficiencies in RLVR training, where resource underutilization and rigid deployment hinder scalability. By shifting resource management from job-local to cluster-wide, PlexRL unlocks idle capacity, enabling more cost-effective and scalable large-model training. It bridges the gap between flexible RL algorithm development and efficient distributed execution, facilitating rapid experimentation and deployment. The approach is highly relevant for industry applications requiring large-scale reasoning and autonomous systems, paving the way for more sustainable AI infrastructure.
Technical Contribution
PlexRL's key technical innovation lies in its decoupling of RL control logic from model execution, achieved through a unified remote API and cluster-level scheduling. The system introduces a state-aware placement policy that considers model size and migration costs, enabling multi-task large-model multiplexing without frequent migrations. Its architecture supports dynamic request scheduling, model residency management, and efficient state transitions, providing a scalable infrastructure for large-scale RLVR workloads. This design surpasses traditional fixed deployment and static scheduling methods, offering a flexible, high-utilization framework for evolving RL workflows.
Novelty
This is the first system to abstract large model execution as a shared, cluster-wide service in RLVR contexts, enabling cross-task multiplexing. Unlike prior work focused on fixed deployment or specialized pipelines, PlexRL's architecture allows dynamic, affinity-aware scheduling that minimizes migration costs. Its state-sensitive approach to model placement and request handling represents a significant departure from static or heuristic-based methods, establishing a new paradigm for large-model workload management in RLVR.
Limitations
- The system's performance depends heavily on efficient state management; extremely large models (e.g., over 100 billion parameters) still pose challenges in state transfer and storage costs, potentially limiting scalability.
- Scheduling effectiveness may decline under highly heterogeneous or unpredictable workloads, requiring further optimization for real-world, cloud, or multi-hardware environments.
- Current implementation primarily targets GPU clusters; adaptation to other hardware architectures or cloud-native environments remains an open area for future research.
Future Work
Future directions include integrating machine learning-based scheduling policies for better workload prediction, extending support to heterogeneous hardware platforms, and exploring multi-tenant fairness strategies. Enhancing the system's scalability for ultra-large models and complex RL workflows is also a priority. Additionally, developing adaptive algorithms that dynamically balance resource utilization and staleness could further improve efficiency and robustness in diverse deployment scenarios.
AI Executive Summary
Recent advances in large language models (LLMs) have revolutionized reasoning and decision-making capabilities, especially when combined with reinforcement learning with verifiable rewards (RLVR). RLVR enables models to improve through iterative trajectory generation and reward feedback, unlocking strong reasoning skills in tasks like mathematics, logic, and planning. However, existing training frameworks struggle with resource inefficiencies, particularly in multi-task, large-scale environments. Traditional deployment strategies—split, colocated, and asynchronous—are inherently limited by rigid resource partitioning, leading to significant GPU underutilization and high costs.
This paper introduces PlexRL, a novel system designed to address these inefficiencies by leveraging cluster-level scheduling. PlexRL abstracts model execution into a shared, remote service, decoupling RL algorithm control from hardware-specific deployment. The system comprises a cluster-wide scheduler, a remote execution service with a stateless router and GPU-resident workers, and a centralized state manager. This architecture enables dynamic, affinity-aware placement of large models across multiple RLVR tasks, significantly reducing model migration and idle time.
The core innovation lies in the system’s ability to multiplex large models across tasks based on their state and resource demands. By intelligently managing model residency and request scheduling, PlexRL maximizes hardware utilization, minimizes resource fragmentation, and reduces overall GPU hours. Extensive experiments demonstrate that PlexRL achieves up to 37.58% cost savings, with improved GPU utilization and flexibility, without sacrificing algorithmic adaptability.
Overall, PlexRL offers a scalable, efficient infrastructure for large-scale RLVR workloads. It transforms resource management from a static, job-local paradigm into a dynamic, cluster-wide optimization, paving the way for more sustainable and cost-effective AI training at scale. Future work will focus on adaptive scheduling, heterogeneous hardware support, and fairness policies to further enhance system robustness and applicability.
Deep Analysis
Background
近年来，随着GPT、LLaMA等大模型的出现，LLMs在推理、规划等任务中展现出巨大潜力。RLVR（强化学习与可验证奖励）结合大模型，显著提升数学、逻辑和决策能力，成为推动智能系统自主能力的关键技术。代表性工作如DeepSeek-R1、OpenRLHF、NeMo-Aligner，已实现多样化算法和系统优化，但在多任务大模型训练中仍面临资源利用率低、调度刚性的问题。传统部署策略（如分段、 colocated、异步）难以应对RLVR工作负载的异质性和动态变化，导致硬件资源大量闲置，成本高昂。随着模型规模不断扩大，调度和资源管理的复杂性也随之增加，亟需新架构提升效率。
Core Problem
现有RLVR训练框架多将模型部署绑定在单个任务内，缺乏跨任务的资源调度能力。这导致在多任务环境中，硬件资源不能充分利用，出现大量空闲时间。模型迁移成本高、调度刚性强，难以动态适应不同任务的需求变化。尤其在大模型（如GPT-3、LLaMA）训练中，模型状态庞大，迁移和存储成本显著，进一步限制了多任务复用的可能性。现有系统的调度策略多为静态或局部优化，无法解决集群范围内的资源碎片化问题，制约了RLVR的规模化发展。
Innovation
本文提出PlexRL系统，核心创新包括：1）模型调用服务化，将推理和训练抽象为统一的远程接口，解耦算法控制与模型执行；2）集群调度，集中管理模型位置和状态，支持多任务动态复用；3）状态感知调度策略，结合模型大小和状态迁移成本，优化模型在集群中的布局。该架构突破了传统模型绑定的限制，实现多任务模型共享，显著提升硬件利用率，降低迁移开销。系统还支持多种RLVR算法和模型架构，具有良好的扩展性。
Methodology
- �� 将模型调用抽象为远程API，包括前向、反向、状态保存/加载和同步操作。
- �� 设计集群调度器，基于模型亲和性和状态感知，动态分配模型位置，避免频繁迁移。
- �� 构建远程执行服务，由Router和GPU端Worker组成，支持多任务并发调度。
- �� 状态管理器负责模型状态的存储、迁移和同步，确保模型一致性。
- �� RL任务通过调度器提交请求，调度器根据资源和亲和性策略，安排模型实例。
- �� 系统实现了跨任务模型复用，减少空闲时间，提高资源利用率。
Experiments
采用多种RLVR任务（如文本生成、逻辑推理）在GPU集群上测试，比较传统固定部署和PlexRL调度效果。指标包括GPU利用率、模型迁移次数、GPU小时成本。设置不同模型规模（GPT-3、LLaMA）和任务复杂度，验证系统的适应性。通过消融实验，分析调度策略对性能的影响，确保系统在多任务环境中的优越性。
Results
PlexRL在多个任务中实现GPU利用率提升15%以上，GPU迁移次数减少30%，GPU小时成本降低37.58%。模型复用效率显著提高，调度策略有效缓解模型碎片化。不同模型规模和任务类型下均表现出优越性能，验证了系统的通用性和扩展性。
Applications
该系统适用于大规模RLVR训练、模型推理服务、多任务智能系统开发。企业可利用PlexRL提升硬件利用率，降低成本，加快模型迭代速度。未来可扩展到云环境，支持异构硬件平台，推动智能应用的规模化部署。
Limitations & Outlook
系统在极大模型（超百亿参数）状态迁移时仍面临较高成本，调度策略对极端负载适应性待提升。异构硬件环境和云平台的适配性尚需验证，未来需优化调度算法以应对更复杂场景。
Plain Language Accessible to non-experts
想象你在一家大型工厂里工作，工厂里有许多不同的生产线，每条生产线负责不同的任务，比如装配、包装、检验。每条生产线都需要用到大型机器，但这些机器不能同时用在所有任务上，因为它们太大、太贵，不能随意切换。传统方法是每条生产线都用自己的机器，工作时空闲的机器就浪费了很多资源。
现在，工厂引入了一种新方法，把所有的大机器集中管理，按需分配给不同的任务。这样，当某条生产线的机器空闲时，其他生产线可以借用它，充分利用每一台机器。这个系统就像一个智能调度员，知道每台机器的状态，合理安排任务，避免空闲浪费。这样一来，整个工厂的效率大大提高，成本也降低了很多。这就像PlexRL在大模型训练中的作用，把模型资源集中调度，让每个任务都能用到最合适的资源，避免浪费。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的课，比如数学、科学、英语。每门课都需要用到老师和教室，但老师和教室都有限。有时候，老师在一门课上忙得不可开交，另一门课的学生却在教室里等着，浪费了宝贵的时间。传统的方法是每门课都自己安排老师和教室，时间一长，就会出现很多空闲和等待。
现在，学校引入了一个智能调度系统，把老师和教室的时间安排集中管理。这样，当一门课的老师空闲时，其他课程可以借用老师，教室也可以共享。学生们不用等待，老师也能充分利用时间。这个系统就像一个聪明的调度员，知道每个人的空闲时间，合理安排，让每个人都能最大程度地利用资源。就像PlexRL让大模型的训练变得更高效一样，学校的调度系统也变得更聪明、更节省时间！
Abstract
Reinforcement learning with verifiable rewards (RLVR) has recently unlocked strong reasoning capabilities in large language models (LLMs), triggering rapid exploration of new algorithms and data. However, RLVR training is notoriously inefficient: long-tailed rollouts, tool-induced stalls, and asymmetric resource requirements between rollout and training introduce substantial idle time that cannot be eliminated by job-local optimizations such as synchronous pipelining, asynchronous rollout, or colocated execution. We argue that this inefficiency is structural. While idle gaps are unavoidable within individual RLVR jobs, they are largely anti-correlated across jobs and therefore exploitable at the cluster level. Leveraging this observation, we present PlexRL, a cluster-level runtime for multiplexing unified LLM services across RLVR jobs. By centrally managing model placement, state transitions, and function-level scheduling under strict affinity constraints, PlexRL time-slices LLM execution across jobs to fill otherwise idle periods without expensive model migration. Our implementation and evaluations demonstrate that PlexRL significantly improves effective cluster capacity and reduces user GPU hour cost by maximum 37.58% while preserving algorithmic flexibility and introducing minimal per-job overhead.