SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
SG-OPD introduces sign-consistency gating and phased teacher sampling, improving mathematical reasoning accuracy by 1.98 and 7.50 points over standard OPD.
Key Findings
Methodology
SG-OPD combines a binary verifier as a trust signal with phased teacher sampling (PTS) to address cold-start trajectory misalignment, and employs sign-consistency gating at the token level to dynamically route advantages based on agreement between the verifier and the teacher. The approach integrates reverse KL objectives, advantage estimation, and multi-granularity signal modulation within a policy gradient framework. During training, verified teacher rollouts are gradually phased out, while token-wise advantages are adjusted according to sign agreement, allowing the student to inherit reliable supervision where the teacher is trustworthy. This multi-layered mechanism ensures robustness in complex reasoning tasks, particularly in mathematical benchmarks, by stabilizing early training and enhancing generalization.
Key Results
- On four mathematical reasoning benchmarks, SG-OPD achieves an average accuracy of 29.53% at 32 samples, outperforming OPD by 1.98 points and ExOPD by 1.54, with a pass@32 score of 59.17%, surpassing OPD by 7.50. Notably, on AIME25, the improvement is 5.00 points, demonstrating its effectiveness in high-difficulty scenarios.
- Ablation studies confirm that the sign-consistency gate broadens the safe extrapolation range, allowing λ to reach 1.8 with stable performance, whereas uniform extrapolation collapses beyond λ=1.25. Phased teacher sampling effectively mitigates cold-start bias, with combined mechanisms yielding additive gains. The training maintains higher entropy and reward, indicating better exploration and stability.
- Overall, SG-OPD significantly advances the state-of-the-art in mathematical reasoning, providing a robust, scalable framework that balances exploration, supervision reliability, and model generalization, with potential for broader NLP applications.
Significance
This work addresses fundamental limitations in on-policy distillation, notably trajectory misalignment and token-level preference conflicts, which hinder performance in complex reasoning tasks. By introducing multi-granularity trust signals and adaptive signal routing, SG-OPD enhances the robustness and efficiency of model compression, enabling large language models to be distilled into smaller, more deployable forms without sacrificing reasoning accuracy. Its success in mathematical benchmarks suggests promising applications in education, scientific research, and AI safety, where reliable reasoning is critical. The approach also opens new avenues for integrating verification signals into reinforcement learning and knowledge distillation frameworks, potentially transforming how models learn from and trust their teachers.
Technical Contribution
The paper's key technical contributions include the integration of a binary verifier as a trust signal at both trajectory and token levels, the design of phased teacher sampling to alleviate cold-start bias, and the development of sign-consistency gating to dynamically route advantages based on local agreement. These innovations extend existing policy gradient and advantage estimation techniques, enabling more stable and reliable distillation in high-stakes reasoning tasks. The framework generalizes the extrapolation strategies used in prior works like G-OPD, introducing adaptive, signal-driven control that improves robustness and performance. The implementation leverages PPO with advantage routing, and the theoretical analysis guarantees improved stability and convergence properties under complex conditions.
Novelty
This research is the first to systematically combine multi-granularity trust signals—phase-based sampling and sign-consistency gating—in on-policy distillation. Unlike prior methods relying solely on global parameters or single signals, SG-OPD dynamically adjusts the influence of teacher supervision at both trajectory and token levels, addressing the core issues of trajectory misalignment and token preference conflict. Its dual-mechanism design provides a new paradigm for robust, fine-grained control in knowledge distillation, setting a new benchmark for reasoning tasks in large language models.
Limitations
- The effectiveness heavily depends on the accuracy of the verifier; misjudgments can introduce noise, especially in ambiguous or noisy data scenarios.
- The added mechanisms increase computational complexity and hyperparameter tuning difficulty, which may limit scalability in very large models or real-time applications.
- Current experiments focus on mathematical reasoning benchmarks; applicability to broader NLP tasks like commonsense reasoning or dialogue generation remains to be validated.
Future Work
Future research could explore multi-modal verification signals, integrate self-supervised learning to improve verifier accuracy, and develop more efficient algorithms to reduce computational overhead. Extending the framework to multi-task settings and real-world applications such as legal or medical AI systems could further demonstrate its robustness and versatility. Additionally, theoretical analysis of convergence and stability under diverse data distributions would strengthen the framework's foundation, paving the way for more reliable AI systems in safety-critical domains.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有一台超级厉害的机器人（大模型），它能做很多复杂的任务，但太大太慢，不能随时随地用。于是，你想用一台小一点的机器人（小模型）来学习大机器的技能。以前的方法就像让小机器人模仿大机器的每一步，但有时候大机器给出的指令不够准确，或者小机器人还不够聪明，容易出错。现在，这个新方法像是给小机器人一个“信任标签”，告诉它哪些指令是真的可靠，哪些可能要小心。刚开始，小机器人会多听大机器的建议，慢慢地它学会后，就会自己做决定。每次它做完题后，这个“信任标签”会帮它判断答案是否靠谱，如果答案正确，就多给它鼓励；如果答案错了，就提醒它注意。这样，小机器人既能学到大机器的好东西，又不会被错误的建议带偏。最终，它变得更聪明、更可靠，能解决更难的问题，比如数学题、逻辑推理等。就像是给小机器人装上了“聪明的指南针”，让它在学习和工作中少走弯路，变得更厉害。
Abstract
On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice: trajectory-level alignment between the student and the teacher, and uniform token-level reliability of the teacher's preferences. We therefore propose Sign-Gated On-Policy Distillation (SG-OPD), which uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.
References (15)
Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Wenkai Yang, Weijie Liu, Ruobing Xie et al.
On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
Wenhao Zhang, Yuexiang Xie, Yuchang Sun et al.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu et al.
Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
Nan Jia, Haojin Yang, Xing-Chen Ma et al.
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Yaxuan Li, Yu-Xin Zuo, Bingxiang He et al.
Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation
Jiaze Li, Hao Yin, Haoran Xu et al.
DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
Zhiwei He, Tian Liang, Jiahao Xu et al.
HybridFlow: A Flexible and Efficient RLHF Framework
Guangming Sheng, Chi Zhang, Zilingfeng Ye et al.
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al.
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou et al.
Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal et al.
Sequence-Level Knowledge Distillation
Yoon Kim, Alexander M. Rush
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Samy Bengio, O. Vinyals, N. Jaitly et al.
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, O. Vinyals, J. Dean
Cited By (8)
When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
On-policy Distillation with Verifiable Reward
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation
Simple-OPD: Demystifying Warm-up for On-policy Distillation
Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
Blockwise Policy-Drift Gating for On-Policy Distillation