Linear Dynamics in the RLVR Training of Large Language Models
RLVR training exhibits strong linear trajectories in parameters and outputs, enabling model extrapolation and 6.1× speedup via low-variance noise filtering.
Key Findings
Methodology
This work conducts trajectory-level analysis across diverse models, algorithms, and configurations, measuring parameter and output evolution via R^2 in linear regression. Theoretical models link high-variance RL signals to low-dimensional stable drift, validated through controlled experiments. Key algorithms include GRPO, REINFORCE++, and GSPO, with teacher-forced evaluation ensuring output accuracy. The approach combines empirical trajectory analysis with mechanistic theory to reveal the origin of linearity and its predictive utility.
Key Results
- Across models like DeepSeek-R1-Distill, Open-Nemotron-1.5B, Qwen3-series, and algorithms such as GRPO, GSPO, REINFORCE++, the R^2 of weight and output trajectories consistently exceeds 0.7, indicating pervasive linearity. This pattern remains robust under varied hyperparameters and training setups.
- Trajectory-based extrapolation achieves performance comparable to continued RL training, with performance improvements averaging 4.2%. Parameter extrapolation accelerates training by 6.1×, effectively bypassing late-stage collapse points, validated on benchmarks like AIME24/25, MATH500, and LiveCodeBench.
- High-variance RL signals act as low-pass filters, constraining updates along stable, low-dimensional directions. Empirical and theoretical analyses confirm that stochastic noise drives the model toward a linear drift, enabling accurate future state prediction.
Significance
This discovery fundamentally advances understanding of RLVR training dynamics, revealing a universal linear regime that can be exploited for efficient model extrapolation. It addresses longstanding challenges in training stability, resource consumption, and controllability of large language models. The ability to predict future states from early trajectories opens new avenues for rapid model deployment, reducing costs and improving robustness, with broad implications for AI research and industry applications.
Technical Contribution
The paper introduces a systematic empirical observation of trajectory linearity in RLVR, supported by a mechanistic theory linking high-variance noise to low-dimensional stable drift. It develops practical extrapolation methods in parameter and output spaces, validated across multiple benchmarks, establishing a new paradigm for training acceleration and model forecasting. The work bridges theoretical insights with engineering techniques, enabling principled performance prediction and resource-efficient training.
Novelty
This is the first comprehensive demonstration that RLVR training trajectories in both parameter and output spaces are highly linear, driven by stochastic noise filtering. It departs from prior work that focused on static weight-space properties or single-step analyses, providing a unified, mechanistically grounded understanding of training dynamics and enabling effective trajectory extrapolation.
Limitations
- The analysis primarily covers specific RL algorithms and model architectures; broader validation in diverse, real-world scenarios is needed. The linear extrapolation accuracy diminishes under extreme noise or non-stationary environments. Additionally, predicting late-stage collapse points remains challenging, requiring further refinement.
Future Work
Future research will explore extending the linearity phenomenon to larger, multimodal models, integrating adaptive extrapolation strategies, and understanding the impact of linear dynamics on model generalization and robustness. Developing theoretical bounds for extrapolation accuracy and applying these insights to other training paradigms could further enhance training efficiency and stability.
AI Executive Summary
Reinforcement learning with verifiable rewards (RLVR) has emerged as a transformative technique for fine-tuning large language models, significantly boosting reasoning and coding capabilities. Despite its success, the internal training dynamics remain poorly understood, especially the evolution of model parameters and outputs over training trajectories. This gap hampers efforts to optimize training efficiency and stability.
In this study, we perform a comprehensive trajectory-level analysis across various models, algorithms, and configurations. We uncover a surprising regularity: both parameters and output log-probabilities evolve in a highly linear manner, with R^2 values exceeding 0.7 in most cases. This linearity persists throughout most of the training process, regardless of model size, architecture, or RL algorithm, indicating a universal phenomenon. The key driver of this behavior is identified as the high-variance, noisy training signals inherent in RLVR, which act as a low-pass filter, guiding the optimization along a stable, low-dimensional drift.
This insight has profound practical implications. By leveraging the linearity, we develop methods for parameter and output extrapolation. Our experiments show that extrapolating parameters from early training steps can match the performance of full RL training while accelerating the process by over six times. Similarly, output-space extrapolation effectively bypasses late-stage model collapse, improving final performance by an average of 4.2%. These techniques demonstrate that the training trajectory’s linear regime is not merely descriptive but highly predictive and actionable.
The findings open new avenues for efficient large-scale model training, deployment, and continual learning. They also challenge the conventional view of nonlinear model training, suggesting that stochastic noise can induce a surprisingly simple and predictable structure. Future work will aim to generalize these results to broader settings, refine extrapolation techniques, and deepen the theoretical understanding of training dynamics, ultimately enabling faster, more stable, and more controllable AI systems.
Deep Analysis
Background
随着大规模预训练模型的崛起，微调技术不断演变，强化学习与验证奖励（RLVR）成为提升模型推理和编码能力的重要手段。早期工作如OpenAI的RLHF，主要关注奖励模型设计与优化，但对训练过程中的动力学缺乏系统理解。近年来，研究逐步揭示参数空间的稀疏性、几何路径和Rank-1主导的线性特性，试图解释模型在推理任务中的高效学习机制。然而，这些研究多停留在微观或宏观层面，缺少对训练轨迹的系统分析。本文通过轨迹级分析，发现参数和输出在训练中表现出高度线性，挑战了Transformer模型非线性的认知，开启了新的理解视角。
Core Problem
尽管RLVR在性能上取得突破，但其内部训练动力学仍是“黑箱”。特别是，模型参数和输出的变化轨迹缺乏系统理解，限制了训练优化和模型外推的潜力。如何解释训练中的线性规律？其机制是否普遍？这些问题关系到模型的可控性、训练效率和未来的可扩展性。现有研究多关注单步或宏观行为，缺少轨迹层面的系统分析，导致对训练稳定性和模型能力演化的理解不足。
Innovation
本文的创新点包括：1）系统性发现RLVR训练中参数与输出的高度线性，R^2值普遍超过0.7；2）提出高方差噪声作为线性动力学的根源，利用低阶滤波机制解释稳定漂移；3）基于线性轨迹，开发参数和输出空间的外推方法，实现训练加速和性能提升；4）结合理论分析，验证线性关系的因果机制，为未来训练优化提供理论支撑。这些创新推动了对大模型训练动力学的深刻理解，开启了轨迹外推的实用新路径。
Methodology
- �� 采集多模型、多算法训练轨迹，测量参数和输出随训练步数的变化。• 利用线性回归分析每个参数和输出的轨迹，计算R^2值评估线性程度。• 构建高方差噪声引导的低阶滤波模型，解释轨迹线性化的机制。• 通过调节噪声水平、优化器和初始化条件，验证线性规律的普适性。• 开发参数和输出的轨迹外推方法，预测未来模型状态。• 在数学推理和编码任务中验证外推效果，确保实用性。
Experiments
涵盖13个配置，包括不同模型规模（1.5B-32B）、架构（Llama、Qwen）、算法（GRPO、GSPO、REINFORCE++）和数据集。采集训练中间点，计算参数和输出的R^2值，验证线性规律。调节噪声水平，观察其对轨迹线性的影响。利用轨迹外推在AIME24/25、MATH500和LiveCodeBench上测试预测性能，比较实际训练模型与外推模型的表现。不同优化器和初始化条件的对比，确保结论的普适性。
Results
参数和输出轨迹的R^2值均超过0.7，表现出强烈线性关系。轨迹外推在多个任务中实现性能几乎等同于连续训练，平均提升4.2%。训练速度提升6.1倍，验证噪声引导的低阶滤波机制。不同模型和配置下，线性特性保持一致，证明其普适性。
Applications
该技术可用于模型快速部署、集成和持续学习。通过轨迹线性化，显著减少训练时间和资源，特别适合资源有限的场景。未来结合自适应外推策略，有望实现更智能的训练调度，推动工业界的AI应用落地。
Limitations & Outlook
分析主要基于特定RL算法和模型架构，需验证在更复杂、多模态环境中的适用性。极端噪声或非平稳环境下，线性外推的准确性可能下降。模型崩溃点的预测仍存在误差，需进一步优化。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂每天都在生产不同的产品。工厂的机器会不断调整操作方式，试图变得更快更好。虽然每次调整都带点随机，但你会发现，工厂的整体生产流程其实变得越来越像一条直线——每次改进都沿着之前的方向前进，没有突然偏离。这是因为工厂的调整受到很多噪声和干扰，但这些噪声实际上帮助工厂集中在一个稳定的方向上，逐步优化。类似地，在大模型的RLVR训练中，参数和输出也表现出这种“沿直线前进”的规律。研究发现，模型的变化轨迹非常线性，甚至可以用简单的线性模型预测未来的状态。这意味着我们可以用少量的早期训练数据，预测模型未来的表现，从而大大节省训练时间。这就像提前知道工厂未来的生产线走向，提前做好准备一样，极大提高了效率和稳定性。
ELI14 Explained like you're 14
想象你在学校里学习新技能，比如弹吉他。刚开始练习时，你会不断尝试不同的弾法，有时候会走偏，但慢慢地，你发现只要坚持沿着某个方向练习，就会变得越来越熟练。其实，虽然每次练习都带点随机，但整体上你的弹奏变得越来越像一条直线——你在不断改进，没有突然走偏。这就像科学家发现的一个秘密：在训练大模型时，参数和输出的变化也像沿着一条直线在前进。研究发现，这种线性关系很强，可以用简单的线性模型预测未来的表现。这样一来，我们就可以用早期的训练结果，提前知道模型未来的表现，节省很多时间和资源。就像你提前知道自己什么时候能弹出一首好歌一样，模型也可以提前预测自己会变得多厉害。这让训练变得更快、更稳定，也更容易控制。未来，这个发现可以帮助我们更快地训练出强大的AI，像提前知道未来一样，提前做好准备！
Abstract
Reinforcement learning with verifiable rewards (RLVR) has driven significant performance gains in reasoning-oriented large language models (LLMs), yet its internal training dynamics remain largely a black box. In this work, we perform a comprehensive trajectory-level analysis of RLVR and uncover a striking regularity: across various model families, RL algorithms, and training configurations, RLVR consistently enters a robust linear regime, where both parameter weights and output log-probabilities, measured rigorously via teacher-forced evaluation, evolve in a highly linear manner ($R^2 > 0.7$). Through controlled experiments and theoretical analysis, we demonstrate that this linearity is not a coincidence, but stems from the high-variance, noisy nature of RLVR training signals, which act as a low-pass filter to concentrate optimization along a stable, low-dimensional drift. Moreover, we show that this linear structure is not merely descriptive but powerfully predictive and actionable. Specifically, weight-space extrapolation matches the performance of standard RL optimization while achieving a 6.1x training speedup through periodic re-grounding. Meanwhile, output-space extrapolation serves as a lightweight intervention that effectively bypasses late-stage model collapse, consistently outperforming standard RL across mathematical and coding benchmarks, with an average performance improvement of 4.2%. Our code is available at https://github.com/Miaow-Lab/RLVR-Linearity.