Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

The evolution of large language models (LLMs) like GPT and BERT has revolutionized NLP, but their high computational costs and deployment challenges motivate model compression and knowledge distillation techniques. Traditional off-policy distillation (KD) trains smaller models by mimicking teacher outputs, but suffers from distribution mismatch and limited adaptability. Reinforcement learning (RL) introduced reward-based optimization, enabling models to learn from feedback signals. OPD emerged as a hybrid, utilizing student-generated trajectories and dense token-level rewards, improving efficiency and performance. Prior work demonstrated multi-task and multi-source fusion capabilities, yet faced limitations in reward design flexibility and reference model selection. This paper builds on these foundations, proposing a unified theoretical framework linking OPD with dense RL, and extending it with reward extrapolation to surpass teacher performance.

Existing knowledge distillation methods, including OPD, are constrained by fixed reward weights and rigid reference models, limiting their ability to improve beyond teacher performance. In multi-task and multi-source scenarios, these limitations hinder effective knowledge fusion and performance gains. Moreover, the challenge of designing rewards that can push models beyond their teacher’s capabilities remains unresolved. The core problem is how to dynamically adjust reward influence and reference model choice to enable models not only to imitate but also to innovate and outperform, especially in complex, multi-source environments where the potential for performance gains is significant but underexploited.

This work introduces several key innovations: 1) Reward extrapolation (λ>1), enabling models to learn beyond teacher boundaries; 2) Flexible reference models, supporting multi-source knowledge fusion; 3) Theoretical proof that reward extrapolation can lead to performance surpassing teachers; 4) Empirical validation demonstrating consistent accuracy improvements across benchmarks. These innovations address the rigidity of prior methods, allowing for adaptive, performance-driven learning. The framework unifies RL and distillation, providing a new perspective on model training that leverages dense token-level rewards and dynamic reward weighting, opening pathways for more intelligent and capable models.

• �� Formalize OPD as a dense RL with a fixed reward weight, then generalize by introducing a reward scaling factor λ and a flexible reference model. • Define the objective as maximizing the expected token-level log-probability, weighted by λ, between the student and a combined target (teacher and reference). • Derive the optimal solution showing that λ>1 encourages the student to learn beyond the teacher’s log-probabilities, effectively performing reward extrapolation. • Use theoretical analysis to demonstrate that this approach can surpass the teacher’s performance boundary. • Implement the framework in experiments with math reasoning and code generation, adjusting λ to observe effects on accuracy and response length. • Incorporate reward correction by choosing the teacher’s pre-RL model as reference in weak-to-strong distillation, further improving results.

Experiments involve distilling a large language model (Qwen3-4B) on math reasoning and code generation tasks, using datasets like DeepMath and Eurus-RL-Code. Teachers are obtained via domain-specific RL, and student models are trained with G-OPD under various λ settings. Evaluation metrics include accuracy and response length, with benchmarks such as AIME24/25, HMMT, HumanEval+, MBPP+, and LiveCodeBench. The experiments compare standard OPD, reward interpolation (λ<1), and reward extrapolation (λ>1), analyzing performance gains, stability, and behavior. Additional ablation studies examine reference model choices and reward correction effects, validating the theoretical predictions.

Reward extrapolation (λ>1) yields consistent accuracy improvements of 5-10% over standard OPD across all benchmarks. In multi-source fusion, the student surpasses all individual teachers, demonstrating the ability to learn beyond the original capabilities. Reward correction using the teacher’s pre-RL model further boosts weak-to-strong distillation performance by 3-5%. Response length analysis shows that higher λ increases output verbosity, aligning with performance gains. The results confirm that reward externalization effectively enhances model capabilities, validating the theoretical framework.

The approach is directly applicable to model compression, multi-task learning, and multi-source knowledge fusion in industry. It enables training smaller models that outperform larger teachers, reducing deployment costs. The framework can be integrated into existing training pipelines for NLP, vision, and multimodal models, facilitating more flexible and powerful AI systems. Long-term, it paves the way for autonomous, self-improving AI capable of continuous knowledge integration and performance enhancement.

Reward extrapolation may cause training instability and requires careful hyperparameter tuning. Using pre-trained teacher models for reward correction increases computational costs, limiting scalability. Over-extrapolation risks overfitting, especially in noisy or complex tasks. The method assumes access to teacher pre-RL models, which may not always be available. Future work must address these issues to improve robustness and efficiency.