Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
VIGOR uses reward variance to adaptively allocate rollouts, reducing sampling by up to 2.3× while maintaining performance.
Key Findings
Methodology
VIGOR employs reward variance as a dynamic priority metric for sample allocation. At each iteration, it computes the reward variance for each prompt’s rollout group, then selectively allocates additional rollouts to prompts with the highest variance. This process concentrates computational effort on the most informative samples without increasing total sample count. The approach is grounded in theoretical analysis showing reward variance controls gradient magnitude, with derivations indicating exponential speedup under Pareto-distributed variance. Implementation involves initial small rollout budgets, iterative variance-based filtering, and multiplicative expansion of selected prompts’ rollouts, all integrated into the GRPO framework.
Key Results
- On mathematical reasoning benchmarks, VIGOR achieves target accuracy with up to 2.3× fewer rollouts, outperforming GRPO, GRESO, and RL-ZVP. In coding tasks, it reaches GRPO’s final full pass rate with 1.49× fewer rollouts and improves average test pass rate by 3.4 points. Across six reasoning benchmarks, VIGOR consistently demonstrates higher efficiency and performance, especially on Qwen2.5-3B, where the efficiency gain is most pronounced.
- Experimental results confirm that reward variance-guided sampling accelerates training, reduces computational costs, and enhances final model quality, validating the theoretical speedup analysis and demonstrating practical benefits in real-world tasks.
Significance
This work addresses a fundamental bottleneck in large-scale reinforcement learning—sample inefficiency—by introducing a theoretically grounded, adaptive rollout allocation strategy. By focusing resources on the most informative samples early in training, VIGOR significantly shortens training times and reduces costs, making high-performance models more accessible. Its ability to dynamically adapt based on reward variance offers a scalable solution applicable across various tasks, from mathematical reasoning to code generation. The approach bridges the gap between theoretical insights and practical efficiency, paving the way for faster, more stable training of large models in industry and academia.
Technical Contribution
VIGOR’s key innovation lies in leveraging reward variance as a real-time, online signal for sample prioritization, supported by rigorous theoretical analysis demonstrating exponential speedup under Pareto-distributed variances. The algorithm integrates variance-based filtering with iterative expansion, avoiding additional computational overhead typical of filtering methods. It provides a convergence guarantee and a closed-form speedup ratio, representing a significant step forward in adaptive sample scheduling for RLVR. This approach opens new avenues for efficient large-scale reinforcement learning, with potential extensions to multi-task and multi-modal settings.
Novelty
VIGOR is the first method to directly utilize reward variance as an online, adaptive criterion for sample allocation in RLVR. Unlike prior approaches relying on static difficulty heuristics or historical signals, it dynamically exploits the natural variability in reward signals during training. Its theoretical foundation and empirical validation distinguish it from existing filtering or difficulty-aware sampling methods, offering a scalable, principled solution that aligns computational effort with sample informativeness in real time.
Limitations
- VIGOR’s effectiveness depends on accurate reward variance estimation; noisy or sparse reward signals may diminish its advantage. In highly unbalanced or reward-sparse scenarios, the variance may not reflect true difficulty, limiting its applicability.
- The method’s performance in multi-task or multi-modal settings remains to be validated, as the reward distribution assumptions may not hold universally.
- While computationally efficient, the iterative variance calculation and filtering introduce some overhead, which could be non-trivial in extremely large datasets or models. Further research is needed to optimize these aspects.
Future Work
Future directions include integrating multi-metric signals such as uncertainty or entropy to enhance sample prioritization robustness. Extending VIGOR to multi-task, multi-modal, and continual learning scenarios could broaden its applicability. Additionally, exploring meta-learning strategies for adaptive parameter tuning, and combining with other efficiency techniques like model pruning or quantization, may further accelerate training while maintaining or improving performance.
AI Executive Summary
The rapid growth of large language models has intensified the need for efficient training methods, especially in reinforcement learning frameworks like RLVR that rely on chain-of-thought reasoning. While methods such as GRPO have demonstrated success, their reliance on generating vast numbers of long reasoning trajectories results in high computational costs and instability. Addressing this bottleneck, the present work introduces VIGOR, a novel approach that leverages reward variance as an online, adaptive signal to guide rollout allocation during training.
VIGOR begins with a small, uniform rollout budget across prompts. During each iteration, it computes the reward variance for each prompt’s rollout group, then selectively allocates additional rollouts to prompts with the highest variance. This process iteratively concentrates computational resources on the most informative samples, significantly reducing the total number of rollouts needed to reach target accuracy. Theoretical analysis shows that reward variance controls the gradient magnitude, and under Pareto-distributed variances, VIGOR achieves an exponential speedup over traditional methods.
Empirical evaluations on mathematical reasoning and coding tasks demonstrate that VIGOR reduces rollout requirements by up to 2.3× while maintaining or improving final performance. In math benchmarks, it reaches target accuracy faster; in coding benchmarks, it attains full pass rates with fewer samples and higher test scores. These results validate the theoretical insights and highlight VIGOR’s potential to make large-scale RL training more resource-efficient.
This advancement offers a scalable, principled solution to the longstanding challenge of sample inefficiency in RLVR, with broad implications for AI research and industry. By focusing on the most informative training samples dynamically, VIGOR paves the way for faster, more stable, and cost-effective development of high-performance reasoning models. Future work will explore multi-metric signals, multi-task settings, and further optimization to extend its benefits across diverse AI applications.
Deep Analysis
Background
近年来，随着大规模预训练模型在自然语言处理中的广泛应用，提升模型推理能力成为研究重点。链式推理（Chain-of-Thought）技术在引导模型进行复杂推理方面取得显著进展，但其训练成本高昂，尤其是在强化学习框架中。RLVR（Reinforcement Learning from Verifiable Rewards）通过自动化奖励信号，增强模型推理能力，减少对人工标注的依赖。GRPO作为一种高效的策略优化算法，通过分组相对奖励提升训练效率，但其依赖大量样本生成，导致计算成本激增。近年来，样本筛选和动态调度策略成为研究热点，旨在在早期阶段集中资源于最具信息的样本，从而提升训练效率，减少无用样本的浪费。
Core Problem
现有方法在强化学习中面临样本效率低、训练成本高的问题。GRPO虽能提升效率，但其生成大量长链推理样本的方式极大增加了计算负担，且在训练早期资源分配不足时，模型难以快速收敛。如何在保证模型性能的同时，减少无效样本的生成，成为提升大规模模型训练效率的关键。特别是在早期训练阶段，资源集中在最具信息的样本尤为重要，否则可能错失优化潜力。这一难题关系到训练成本、时间和模型最终性能的平衡。
Innovation
VIGOR的创新点在于：1）引入奖励方差作为样本优先级指标，动态调节轮次分配，避免无效采样；2）结合理论分析，证明奖励方差控制梯度幅度，提供指数级加速保证；3）采用逐步筛选高方差样本、扩展轮次预算的策略，有效集中计算资源。不同于传统静态采样或基于历史信号的方法，VIGOR在训练过程中实时利用奖励方差信号，提升样本利用率和训练效率。其创新在于：理论基础扎实、算法简单高效、适应性强，能在不同任务中实现显著效率提升。
Methodology
- �� 初始化：每个样本设定少量轮次，建立空轮次集。
- �� 生成：在每轮中，针对当前样本集，采样轮次并合并奖励。
- �� 计算：计算每个样本的奖励方差，作为信息量指标。
- �� 筛选：保留最高方差的样本子集，扩展其轮次预算。
- �� 迭代：重复上述步骤，逐步集中资源于最具信息的样本，直到预算用尽。
- �� 最终：利用所有采样样本进行GRPO更新，提升训练效率。
Experiments
在数学推理和编码任务中，采用Qwen系列模型（2.5-1.5B、3B、7B）及Phi-4-Mini-Instruct，基于Hendrycks的MATH数据集和LiveCodeBench v6。对比基线包括GRPO、GRESO和RL-ZVP，评估指标为目标准确率、全通过率和平均测试通过率。超参数设定：T=4轮，初始轮次m0=2，扩展比γ=2，筛选比例α=0.5。实验验证VIGOR在不同模型规模下显著降低轮次需求，提升训练效率。
Results
VIGOR在数学任务中，用最高2.3×轮次节省达成目标，且在Qwen2.5-3B模型上表现出最优效率。在编码任务中，达到GRPO最终全通过率的同时，轮次减少1.49×，平均测试点提升3.4点。多项基准测试显示，VIGOR不仅节省计算资源，还提升了模型最终性能，验证其在实际训练中的优越性。
Applications
该方法适用于大规模推理模型训练，尤其在资源有限或追求高效率的场景。可应用于自动化推理、代码生成、数学问题解决等领域，帮助研究者和企业降低成本，加快模型开发速度。未来，结合多模态数据和多任务学习，VIGOR有望推动AI系统的普及和智能化水平提升。
Limitations & Outlook
VIGOR依赖奖励方差的准确估计，在奖励信号噪声大或分布偏离假设时效果可能下降。算法在极端稀疏奖励或不平衡场景下表现有限，且在超大模型和复杂任务中的适应性仍需验证。未来需结合多指标、多任务策略，增强鲁棒性和泛化能力。
Plain Language Accessible to non-experts
想象你在厨房里做饭，每次准备食材都要花时间。传统做法是每次都准备很多食材，浪费很多时间和材料。而VIGOR就像厨师根据每次尝试的结果，判断哪些食材最有用，只准备那些最能提升菜肴味道的部分。它通过观察每次尝试的变化，逐步集中在最重要的食材上，节省了时间和资源。这样，厨房里的厨师可以更快做出美味的菜肴，而不用浪费在无用的步骤上。这个策略在训练AI模型时也一样，VIGOR通过奖励的变化来判断哪些训练样本最有价值，集中资源学习，从而更快更好地提升模型能力。
ELI14 Explained like you're 14
想象你在玩一款游戏，每次你都试不同的策略来赢得比赛。传统的方法是每次都试很多不同的策略，不管它们是否有效，这样既花时间又浪费精力。VIGOR就像一个聪明的朋友，他会观察每次尝试的结果，发现哪些策略最容易带来胜利，然后专注于那些策略，反复练习。这样，你就能用更少的时间学会赢得比赛。用在AI训练上，VIGOR也是这样，它会观察每个训练样本带来的奖励变化，优先学习那些能带来最大提升的样本。这样，模型可以更快变得聪明，花更少的时间就能达到目标，就像你变成游戏高手一样！
Abstract
Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3$\times$ fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49$\times$ fewer rollouts, and improves the coding average test pass rate by 3.4 points.