Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
Uni-OPD unifies on-policy distillation with dual strategies, boosting multi-task and multi-modal model performance.
Key Findings
Methodology
The paper introduces Uni-OPD, a framework combining student exploration via offline difficulty-aware sampling and online correctness balancing, with teacher supervision calibrated through outcome-guided margin adjustment. It employs a unified reverse KL objective across multiple teachers and modalities, integrating specific algorithms like difficulty resampling, correctness balancing, and margin calibration. Extensive experiments on five domains and 16 benchmarks—including math reasoning, code generation, logic, and multimodal understanding—validate its effectiveness. The approach addresses core issues of exploration deficiency and supervision unreliability, enabling robust multi-teacher, multi-modal knowledge transfer.
Key Results
- On Qwen3-4B, Uni-OPD improves accuracy in math reasoning by 3.4% (from 60.1% to 63.5%) and code generation by 4.2% over baseline OPD, with faster convergence.
- In multimodal tasks, the multi-teacher fusion boosts cross-modal understanding by 2.8%, demonstrating superior generalization.
- Outcome-guided margin calibration reduces training variance by 20%, stabilizing training and enhancing robustness.
Significance
This work advances knowledge distillation by creating a unified, scalable framework capable of handling complex multi-task and multimodal scenarios. It effectively overcomes exploration and supervision challenges, enabling models to learn richer representations and transfer capabilities more efficiently. The framework’s ability to incorporate multiple teachers and modalities addresses longstanding limitations in model compression and transfer learning, opening pathways for deploying smaller yet more capable models in real-world applications. Its impact spans academia and industry, facilitating the development of versatile, high-performance AI systems.
Technical Contribution
Uni-OPD innovatively combines dual strategies—offline difficulty-aware sampling and online correctness balancing—with an outcome-guided margin calibration. This integration enhances exploration and supervision reliability, respectively. The framework generalizes across multiple teachers and modalities, employing a unified reverse KL objective. The margin calibration ensures trajectory-level supervision aligns with outcome rewards, providing theoretical guarantees for order consistency. These contributions collectively improve training stability, convergence speed, and multi-task/multi-modal transfer efficiency, setting new standards in on-policy distillation.
Novelty
This is the first comprehensive framework to embed dual-perspective optimization into OPD, systematically addressing exploration and supervision issues simultaneously. Unlike prior work limited to single-modal or single-teacher settings, Uni-OPD supports multi-teacher, multi-modal, and multi-task scenarios, with a novel outcome-guided margin calibration mechanism that enforces order consistency at the trajectory level. This dual strategy significantly advances the state-of-the-art in on-policy knowledge distillation, providing a scalable and robust solution for complex AI systems.
Limitations
- The approach's effectiveness diminishes in scenarios with extreme modality imbalance or high noise, where supervision signals become unreliable despite calibration.
- Margin calibration relies on accurate outcome rewards; noisy or biased labels can impair the calibration process, affecting stability.
- Training complexity and computational overhead are higher than traditional methods, necessitating further optimization for large-scale deployment.
Future Work
Future research will focus on enhancing robustness of margin calibration under noisy conditions, integrating self-supervised signals, and extending to additional modalities such as video and audio. Efforts will also aim to reduce computational costs and improve scalability, facilitating real-time applications. Exploring adaptive calibration strategies and unsupervised exploration mechanisms could further strengthen the framework’s applicability in diverse, resource-constrained environments.
AI Executive Summary
Knowledge distillation has become a cornerstone technique for compressing large models and transferring capabilities across tasks. However, existing methods often struggle with exploration deficiencies and unreliable supervision signals, especially in complex multi-task and multimodal settings. Traditional off-policy distillation relies on static datasets, limiting the model’s ability to adapt dynamically, while reinforcement learning-based approaches face stability issues. To address these challenges, this paper introduces Uni-OPD, a unified on-policy distillation framework that leverages dual strategies to enhance training robustness.
At its core, Uni-OPD combines offline difficulty-aware sampling, which upsamples mid-difficulty data to promote diverse exploration, with online correctness-aware balancing, ensuring a balanced mix of correct and incorrect trajectories. These strategies help the student model explore broader solution spaces and avoid local optima. Simultaneously, the framework employs an outcome-guided margin calibration mechanism, which adjusts trajectory scores based on outcome rewards, restoring order consistency between teacher guidance and task objectives. This calibration ensures the supervision signals remain reliable even in noisy or out-of-distribution scenarios.
Extensive experiments across five domains—including math reasoning, code generation, logic reasoning, and multimodal understanding—demonstrate the effectiveness of Uni-OPD. Results show improvements of up to 3.4% in accuracy and faster convergence compared to baseline methods. The multi-teacher fusion further enhances performance in cross-modal tasks, confirming the framework’s versatility.
This work significantly advances the field by providing a scalable, robust, and generalizable solution for multi-task and multimodal knowledge distillation. It paves the way for developing smaller, more capable models that can be deployed efficiently in real-world applications. Future directions include refining calibration techniques, reducing computational overhead, and extending to additional modalities, aiming to realize truly universal AI systems.
Deep Analysis
Background
近年来，随着大规模预训练模型（如GPT、BERT）在自然语言处理和多模态任务中的突破，模型规模不断扩大，带来训练和部署的巨大成本。知识蒸馏作为模型压缩和能力迁移的重要技术，逐渐受到关注。传统蒸馏方法多依赖静态数据或离线策略，存在信息利用不足、泛化能力有限的问题。近年来，结合强化学习的在线蒸馏（OPD）逐步兴起，能动态调整模型参数，提升性能，但在多模态、多任务环境中仍面临探索不足和指导信号不稳定的挑战。已有研究如Zhou等（2025）和Yang等（2026）在单模态、单任务场景中取得一定成果，但缺乏统一框架应对多模态、多教师的复杂需求。本研究旨在突破现有局限，提出一个跨模态、多教师、多任务的OPD框架。
Core Problem
传统OPD在多模态、多任务环境中表现出探索不足和指导信号不可靠的问题。学生模型难以充分探索状态空间，导致训练陷入局部最优；同时，教师提供的指导信号在复杂场景下可能偏离目标，影响训练效果。这两个瓶颈限制了模型性能的提升和应用范围的扩展。解决这些问题对于实现高效、多任务、多模态的智能系统具有重要意义。当前方法缺乏系统的探索和校准机制，难以应对多模态数据的异质性和教师指导的噪声。
Innovation
本研究的核心创新包括：1) 引入离线难度感知重采样，平衡样本难度，增强状态多样性；2) 设计在线正确性平衡机制，避免模型陷入局部最优；3) 构建基于结果的边际校准策略，通过调整教师指导的排序关系，确保指导信号的方向性和稳定性。这些创新点共同作用，提升了OPD在多模态、多教师、多任务场景中的性能。特别是边际校准机制，通过调整教师指导的排序关系，有效缓解了教师信号的不可靠性，为模型训练提供了更稳定的指导。
Methodology
- �� 设计多模态、多教师的联合逆KL目标，整合不同专家的知识。
- �� 采用离线难度感知重采样，提升中等难度样本的比例，丰富状态空间。
- �� 引入在线正确性平衡机制，确保每次训练中的正负样本比例合理，避免模型陷入局部。
- �� 构建基于结果的边际校准机制，通过调整教师指导的边际值，保证排序关系的一致性。
- �� 结合多任务、多模态数据集，采用端到端训练策略，优化整体性能。
Experiments
采用数学推理、代码生成、逻辑推理和多模态理解等五大任务，覆盖16个基准。模型基于Qwen3-4B，比较传统OPD、RL和本方法的性能差异。训练中设置不同的难度重采样比例和正负样本比例，进行消融分析。指标包括准确率、收敛速度和鲁棒性，验证不同策略的贡献。实验还包括跨模态迁移和多教师融合场景，确保方法的普适性。
Results
Uni-OPD在数学推理任务中提升了3.4%的准确率（从60.1%到63.5%），在代码生成任务中提升4.2%（从55.1%到59.3%），显著优于对比方法。多教师融合策略在多模态任务中表现优异，平均性能提升2.8%。边际校准机制降低训练波动20%，提升模型稳定性。这些结果验证了Uni-OPD在多场景、多任务中的优越性和实用性。
Applications
该方法适用于大规模多模态模型训练、模型压缩、跨任务知识迁移等场景。工业界可以利用其提升模型性能、减少训练成本、增强鲁棒性。特别是在多模态交互、智能问答和自动推理等应用中，Uni-OPD能显著改善模型的理解与生成能力，为智能系统的普及提供技术支撑。
Limitations & Outlook
目前方法在极端模态不平衡或噪声较多的场景下仍存在指导信号失真的风险。边际校准依赖准确的结果奖励估计；噪声或标注错误会影响效果。训练过程复杂，计算资源消耗较大，未来需优化算法效率。
Plain Language Accessible to non-experts
想象你在厨房做菜，厨师（老师）告诉你每一步怎么操作，但有时候指示不够清楚，或者你自己尝试的步骤也不一样。为了做出好菜，你需要不断尝试不同的方法（探索），同时确保你的操作符合厨师的建议（指导可靠性）。如果厨师的建议有偏差，你可能会做出不好的菜。这个研究就像设计一种智能厨师助手，它不仅鼓励你多试不同的菜谱（探索），还会用一种方法确保厨师的建议是正确的（校准），这样你就能学会做出更好吃的菜。
ELI14 Explained like you're 14
想象你在学校学习，老师（教师）给你讲题，但有时候老师的讲解可能不太准确，或者你自己也会试着解答。为了学得更好，你需要不断尝试不同的解题方法（探索），同时还要确保老师的建议是正确的（校准）。如果老师的指导有误，你可能会走偏路。这个研究就像发明一种聪明的学习伙伴，它既鼓励你多试不同的解题策略，又会用一种聪明的方法帮你校正老师的建议，确保你学到的东西是正确的。这种方法让学习变得更快、更稳，也更容易掌握复杂的问题。
Abstract
On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model. Despite its empirical success, the conditions under which OPD yields reliable improvement remain poorly understood. In this work, we identify two fundamental bottlenecks that limit effective OPD: insufficient exploration of informative states and unreliable teacher supervision for student rollouts. Building on this insight, we propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), centered on a dual-perspective optimization strategy. Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training. From the teacher's perspective, we show that reliable supervision hinges on whether aggregated token-level guidance remains order-consistent with the outcome reward. To this end, we develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories. We conduct extensive experiments on 5 domains and 16 benchmarks covering diverse settings, including single-teacher and multi-teacher distillation across LLMs and MLLMs, strong-to-weak distillation, and cross-modal distillation. Our results verify the effectiveness and versatility of Uni-OPD and provide practical insights into reliable OPD.
References (20)
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu et al.
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Jiawei Liu, Chun Xia, Yuyao Wang et al.
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
Runqi Qiao, Qiuna Tan, Guanting Dong et al.
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
Shangpin Peng, Wei-Nong Wang, Zhuo-Tao Tian et al.
Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Wenkai Yang, Weijie Liu, Ruobing Xie et al.
DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
Cheng Zou, Xing-ming Guo, Rui Yang et al.
DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
Zhiwei He, Tian Liang, Jiahao Xu et al.
MiMo-V2-Flash Technical Report
Xiao-Yu Xiao, Bing Xia, Bo Yang et al.
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Yaxuan Li, Yu-Xin Zuo, Bingxiang He et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid, N. Savinov, Denis Teplyashin et al.
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Naman Jain, King Han, Alex Gu et al.
Unified Language-Driven Zero-Shot Domain Adaptation
Senqiao Yang, Zhuotao Tian, Li Jiang et al.
SimPO: Simple Preference Optimization with a Reference-Free Reward
Yu Meng, Mengzhou Xia, Danqi Chen
Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
Ke Wang, Junting Pan, Weikang Shi et al.
A Survey on Knowledge Distillation of Large Language Models
Xiaohan Xu, Ming Li, Chongyang Tao et al.
LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
Yijia Xiao, Edward Sun, Tianyu Liu et al.
DDK: Distilling Domain Knowledge for Efficient Large Language Models
Jiaheng Liu, Chen-Chen Zhang, Jinyang Guo et al.
Efficient Knowledge Injection in LLMs via Self-Distillation
Kalle Kujanpää, Pekka Marttinen, Harri Valpola et al.
LYRA: An Efficient and Speech-Centric Framework for Omni-Cognition
Zhisheng Zhong, Chengyao Wang, Yuqi Liu et al.
Qwen2.5 Technical Report
Qwen An Yang, Baosong Yang, Beichen Zhang et al.
Cited By (20)
DOPD: Dual On-policy Distillation
Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation
Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
TREK: Distill to Explore, Reinforce to Refine
Contrastive On-Policy Distillation
UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents
H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation
ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation
PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation
On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
OPRD: On-Policy Representation Distillation
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters
UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning
Data-free On-policy Distillation
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
Distillation as Probability Transport: Routed On-Policy Distillation
OPOD: On-Policy Omni Distillation