On-Policy Supervised Fine-Tuning for Efficient Reasoning
Simplified on-policy supervised fine-tuning (SFT) with truncation-based length reward improves reasoning efficiency and training speed, outperforming complex RL methods.
Key Findings
Methodology
This paper analyzes the misalignments in multi-reward frameworks, removing KL regularization and group normalization, replacing them with a length truncation reward. Self-generated data filtering for correctness and conciseness enables supervised fine-tuning (SFT). Extensive experiments on five math benchmarks demonstrate that this approach reduces chain-of-thought (CoT) length by up to 80%, maintains or improves accuracy, and enhances training efficiency—halving GPU memory and accelerating convergence by 70%. The method emphasizes principled simplicity over complex reward shaping.
Key Results
- On DeepSeek-R1-1.5B, on-policy SFT achieves 59.9% accuracy and Pass@N of 73.6%, with an 80% reduction in average token length (from 10,178 to 2,186). It surpasses RL-based methods in the accuracy–efficiency Pareto frontier (Eff=2.74% vs. 2.55%).
- Training efficiency improves significantly: GPU memory usage drops by 50%, convergence time shortens by 70%. The model consistently maintains high accuracy across various length budgets, demonstrating robustness.
- Removing KL regularization and group normalization simplifies the optimization, yet achieves state-of-the-art performance, confirming the effectiveness of the minimalist design.
Significance
This work challenges the necessity of complex reward engineering in large reasoning models, proposing a simple, principled alternative that reduces training costs and improves inference efficiency. It addresses core bottlenecks in deploying large models in resource-constrained environments and offers a new perspective on reward design, emphasizing clarity and simplicity. The findings suggest that future progress may rely more on elegant, theoretically sound strategies than on increasingly elaborate reward schemes, potentially transforming the development and application of large-scale AI systems.
Technical Contribution
The main contribution lies in the theoretical and empirical demonstration that removing KL regularization and group normalization, combined with a length truncation reward, transforms the multi-reward RL framework into a straightforward supervised fine-tuning process. This approach leverages on-policy data filtering, ensuring stable training and superior performance. It bridges the gap between reinforcement learning and supervised learning, providing a new paradigm for efficient reasoning model training with lower computational overhead.
Novelty
This is the first systematic attempt to replace complex multi-reward RL objectives with a minimalist, truncation-based reward coupled with on-policy supervised fine-tuning. It fundamentally redefines the optimization paradigm for efficient reasoning, emphasizing simplicity and principled design over reward engineering complexity, and demonstrates superior results across multiple benchmarks.
Limitations
- The approach relies on the verifiability of correctness and length, limiting its applicability to tasks where responses are objectively measurable. It may struggle with subjective or ambiguous tasks.
- Choice of length threshold τ is critical; too short may impair performance, too long reduces efficiency gains.
- In highly complex or multi-modal tasks, additional mechanisms may be necessary to maintain generalization and robustness.
Future Work
Future directions include developing adaptive length thresholds, extending the framework to multi-task and multi-modal settings, and exploring theoretical foundations for reward simplification. Further research may also focus on integrating this approach into real-world applications such as automated reasoning, education, and decision support systems, aiming for broader impact and scalability.
AI Executive Summary
Large reasoning models (LRMs) have revolutionized AI's ability to perform complex, multi-step reasoning tasks, yet their training remains computationally intensive and often unstable. Reinforcement learning (RL), especially algorithms like Group Relative Policy Optimization (GRPO), has been employed to enhance reasoning capabilities by optimizing long chain-of-thought (CoT) trajectories. However, these methods introduce complex reward structures, involving multiple signals such as correctness and brevity, along with mechanisms like KL regularization and group normalization, which often destabilize training and lead to suboptimal trade-offs.
This paper critically reevaluates the necessity of such complexities, proposing a minimalist yet effective alternative. By removing KL regularization and group normalization, and replacing multi-reward objectives with a simple truncation-based length penalty, the authors reformulate the optimization as a supervised fine-tuning (SFT) process on self-generated data filtered for correctness and conciseness. This on-policy SFT approach leverages the verifiability of answers and length, avoiding the pitfalls of reward unreliability and normalization ambiguities.
Extensive experiments across five mathematical reasoning benchmarks demonstrate that this simplified method achieves state-of-the-art accuracy–efficiency trade-offs. Specifically, it reduces CoT length by up to 80%, maintains or slightly improves accuracy, and surpasses RL-based methods in efficiency metrics. Additionally, training efficiency improves dramatically, with GPU memory usage halved and convergence time reduced by 70%. The results highlight that principled simplicity can outperform complex RL strategies, providing a scalable, stable, and resource-efficient training paradigm.
Overall, this work advocates for a paradigm shift towards minimalistic, theoretically grounded training strategies in large reasoning models. It opens avenues for deploying high-performance models in resource-constrained settings and underscores the importance of aligning optimization objectives with task properties. Future research will explore adaptive thresholds, multi-task generalization, and industrial applications, aiming to further democratize and accelerate AI reasoning capabilities.
Deep Analysis
Background
近年来，大规模推理模型（LRMs）在自然语言处理领域取得突破，代表性工作如Group1的GRPO算法通过强化学习（RL）实现长链推理能力。此类模型在数学推理、问答等任务中表现优异，但训练成本高、过程不稳定，限制了其实际应用。为解决这一问题，研究者尝试引入多奖励机制，优化推理路径长度，提升效率，但复杂奖励设计带来训练不稳定、性能波动。传统监督微调（SFT）虽简洁，但难以兼顾推理速度与准确性。本文在此背景下，重新审视奖励设计的合理性，提出更简洁高效的训练策略。
Core Problem
现有多奖励强化学习方法在提升推理效率方面虽取得一定成效，但因奖励机制复杂、训练不稳定，难以广泛应用。特别是在多奖励场景中，奖励冲突和归一化机制引入偏差，限制模型性能。如何在保证推理准确率的同时，显著缩短推理路径，降低训练成本，成为亟待解决的问题。本文旨在通过简化奖励设计，提升训练稳定性和效率，推动大规模推理模型的实用化。
Innovation
核心创新包括：1）剥离KL正则和分组归一化机制，简化优化目标；2）引入基于长度截断的二元奖励，直接过滤超长或错误响应；3）利用自生成数据进行监督微调，保持模型性能同时提升效率。这些创新突破了复杂奖励框架的局限，强调原则性简洁，提升训练稳定性和泛化能力。
Methodology
- �� 构建自生成数据过滤机制，筛选正确且简洁的推理路径；• 移除KL正则，采用纯监督微调目标；• 设计长度截断奖励，定义最大长度阈值，超长响应获得零奖励；• 采用on-policy策略，动态采样响应，确保数据分布一致；• 训练流程包括数据采样、过滤、梯度计算和模型更新，确保训练稳定高效。
Experiments
在DeepScaleR数据集上，使用DeepSeek-R1-1.5B和7B模型，比较RL、SFT和本文方法。指标包括准确率、推理长度、效率（Eff）等。五个数学基准（GSM8K、MATH-500、AMC23、AIME24、AIME25）涵盖不同难度和数据规模。超参数设置包括长度阈值、采样温度、训练轮次，进行消融验证。实验还分析不同阈值和采样策略对性能影响。
Results
实验显示，on-policy SFT在五个基准上均优于复杂RL方法，推理长度缩短最高80%，准确率保持或略优（如59.9%对比59.0%）。效率指标显著提升，GPU内存减半，收敛时间缩短70%。在不同长度预算下，模型始终处于效率-准确的Pareto边界，验证其鲁棒性。消融验证确认去除复杂奖励机制的有效性，简洁策略即能达成优异性能。
Applications
该方法适用于自动问答、智能客服、教育辅导等场景，尤其在资源有限环境中实现高效推理。只需确保答案的可验证性，模型即可快速部署。未来结合多任务学习和自适应长度调节，有望扩展到多模态推理和复杂决策支持，推动行业智能化升级。
Limitations & Outlook
该策略依赖答案的可验证性，难以应用于主观或模糊任务。长度阈值选择敏感，过短影响性能，过长降低效率。在极端复杂或多模态任务中，仍需结合其他机制以保证泛化。未来需研究自适应阈值和多奖励机制的理论基础，提升模型鲁棒性和泛用性。
Plain Language Accessible to non-experts
想象你在厨房做饭，目标是做出既好吃又快的菜。传统做法像用很多调料和步骤，虽然味道好但耗时。现在，我们用一种简单的方法：只关注菜是否够香（正确）和做得快（简洁）。只要菜符合这两个条件，就算成功。这样厨师（模型）不用复杂的调料，只用最基本的原则，快速做出满意的菜。这就像用一个简单的计时器和味道检测器，直接筛选出最优的菜肴，节省时间又保证质量。这个方法让厨房效率大大提升，菜也变得更快、更好吃。
ELI14 Explained like you're 14
想象你在学校写作业，有时候题目很难，老师会告诉你：只要答对就行，不用写太长。以前老师会给很多提示，比如字数、用词，确保你没跑题。但这样反而让作业变得复杂，还可能出错。现在，老师只关心你答对了没有，写得是不是简洁明了。只要答对，答案不长，就算完成。这就像用一个简单的评分标准，只看答案对不对和字数是否合适，不需要复杂的评分规则。这样一来，你写作业就更快、更轻松，还能保证质量。这种方法就像用最简单的规则，让你专注在最重要的事情上，效率更高，压力更小。
Abstract
Large reasoning models (LRMs) are commonly trained with reinforcement learning (RL) to explore long chain-of-thought reasoning, achieving strong performance at high computational cost. Recent methods add multi-reward objectives to jointly optimize correctness and brevity, but these complex extensions often destabilize training and yield suboptimal trade-offs. We revisit this objective and challenge the necessity of such complexity. Through principled analysis, we identify fundamental misalignments in this paradigm: KL regularization loses its intended role when correctness and length are directly verifiable, and group-wise normalization becomes ambiguous under multiple reward signals. By removing these two items and simplifying the reward to a truncation-based length penalty, we show that the optimization problem reduces to supervised fine-tuning on self-generated data filtered for both correctness and conciseness. We term this simplified training strategy on-policy SFT. Despite its simplicity, on-policy SFT consistently defines the accuracy-efficiency Pareto frontier. It reduces CoT length by up to 80 while maintaining original accuracy, surpassing more complex RL-based methods across five benchmarks. Furthermore, it significantly enhances training efficiency, reducing GPU memory usage by 50% and accelerating convergence by 70%. Our code is available at https://github.com/EIT-NLP/On-Policy-SFT.