When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
RA-OPD improves student model performance by filtering misaligned trajectories, excelling in math and code benchmarks.
Key Findings
Methodology
RA-OPD filters out trajectories whose distillation returns are inconsistent with outcome rewards, enhancing student model performance without additional computational cost. It operates directly on student-generated trajectories.
Key Results
- RA-OPD significantly outperforms standard OPD on seven math and three code benchmarks with over 5% average improvement.
- On the Qwen3-8B-Base model, RA-OPD improved AIME24 benchmark by 8.75 percentage points.
- In code generation tasks, RA-OPD's average pass@4 increased by 1.51 percentage points.
Significance
RA-OPD addresses the issue of misaligned teacher guidance, significantly improving model accuracy and reliability. This method has important implications in academia and industry, especially in training large models requiring high precision and reliability.
Technical Contribution
Building on existing OPD methods, RA-OPD introduces a new reward alignment mechanism, providing a novel trajectory filtering strategy to avoid the negative impact of misaligned teacher guidance.
Novelty
RA-OPD is the first to propose reward alignment at the trajectory level, offering more comprehensive guidance consistency compared to existing token-level methods.
Limitations
- RA-OPD may filter out some correct trajectories early in training, affecting initial learning progress.
- The method relies on accurate outcome verifiers, which may be challenging in complex tasks.
Future Work
Future research could explore applying RA-OPD to more tasks and models, and combine it with other optimization techniques to further enhance model performance.
AI Executive Summary
In training large language models, teacher guidance often misaligns with student model outcome rewards, leading to performance degradation. RA-OPD addresses this by filtering out trajectories whose distillation returns are inconsistent with outcome rewards, thereby enhancing student model performance.
The method was evaluated on math and code benchmarks using the Qwen3 and DeepSeek-R1 model families, showing significant improvements over standard OPD and other variants, particularly in complex math reasoning tasks.
RA-OPD not only improves model accuracy but also maintains training efficiency comparable to standard OPD, offering a new approach for efficient training of large models. Future research could further optimize this method and explore its potential applications in other domains.
Deep Analysis
Background
In recent years, large language models (LLMs) have made significant strides in natural language processing. Model distillation, an effective knowledge transfer method, is widely used to transfer the capabilities of large teacher models to smaller student models. However, traditional distillation methods rely on teacher guidance on student-generated prefixes, which may misalign with the final rewards of the student model, leading to performance degradation.
Core Problem
During OPD, teacher models may provide guidance that misaligns with outcome rewards, causing the student model to optimize towards incorrect trajectories. This misalignment is a major cause of performance degradation, especially in tasks requiring high precision, such as math reasoning and code generation.
Innovation
RA-OPD introduces a reward alignment mechanism to address the issue of misaligned teacher guidance. It aligns trajectories at the trajectory level, filtering out inconsistent trajectories to enhance student model performance. Unlike existing methods, RA-OPD requires no additional computational cost and operates directly on student-generated trajectories.
Methodology
- �� RA-OPD applies a reward alignment check to each sampled trajectory.
- �� Computes the distillation return of the trajectory and compares it with the outcome reward.
- �� Filters out inconsistent trajectories, retaining consistent ones for optimization.
- �� The method is applied directly to student-generated trajectories without requiring additional student rollouts or teacher evaluations.
Experiments
RA-OPD was evaluated on seven math and three code benchmarks using the Qwen3 and DeepSeek-R1 model families. The experiments used DAPO-Math-17K and Eurus-2-RL-Data-Code-25K datasets, comparing various methods including standard OPD, ExOPD, and Uni-OPD.
Results
RA-OPD excelled across all tested benchmarks, particularly in math reasoning tasks, with an average improvement of over 5%. In code generation tasks, RA-OPD's average pass@4 increased by 1.51 percentage points, demonstrating its broad applicability across different tasks.
Applications
RA-OPD is applicable to large-scale model training requiring high precision and reliability, such as automated code generation and complex math reasoning. The method can improve model accuracy and stability without increasing computational costs.
Limitations & Outlook
RA-OPD may filter out some correct trajectories early in training, affecting initial learning progress. Additionally, the method relies on accurate outcome verifiers, which may be challenging in complex tasks. Future research could explore combining RA-OPD with other optimization techniques to further enhance model performance.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen with a teacher guiding you. Sometimes, the teacher might give wrong advice, like adding too much salt. RA-OPD acts like a smart assistant that checks if each step aligns with the final taste goal. If the teacher's advice is off, the assistant alerts you to ignore it, ensuring your dish tastes better in the end.
ELI14 Explained like you're 14
Imagine you're playing a game with a teacher telling you where to go, but sometimes the teacher points you in the wrong direction. RA-OPD is like a smart helper that checks if the teacher's advice really helps you win the game. If the advice is wrong, the helper tells you to ignore it, so you can finish the game faster!
Glossary
On-Policy Distillation (OPD)
A model distillation method where the student model receives guidance from the teacher model while generating prefixes.
Used to transfer knowledge from teacher models to student models.
Reward-Aligned On-Policy Distillation (RA-OPD)
Filters out trajectories whose distillation returns are inconsistent with outcome rewards.
Key method for improving student model performance.
Trajectory
The complete sequence of outputs generated by a model given an input.
Used to evaluate whether the generated path aligns with rewards.
Distillation Return
The distillation return of a trajectory, used to measure the effectiveness of teacher guidance on the student model.
Used to determine if a trajectory aligns with outcome rewards.
Outcome Reward
The reward for the model's generated result, typically used to evaluate the correctness of the model's output.
Used to compare with the trajectory's distillation return.
Open Questions Unanswered questions from this research
- 1 How to implement accurate outcome verifiers in more complex tasks to ensure RA-OPD's effectiveness.
- 2 How RA-OPD handles reward alignment issues in multi-task learning.
Applications
Immediate Applications
Automated Code Generation
RA-OPD can enhance the accuracy and stability of code generation models, suitable for automation tools in software development.
Long-term Vision
Intelligent Education Systems
By improving reasoning capabilities, RA-OPD can be applied in personalized education systems to provide more accurate learning recommendations.
Abstract
On-policy distillation (OPD) has recently emerged as a popular post-training paradigm for large language models (LLMs), providing an efficient way to transfer the knowledge and capabilities of teacher models into student models. However, teacher guidance on student-generated prefixes is not always reliable. Training should optimize the model to generate responses that are more likely to be correct, or equivalently, to get higher outcome rewards. But during OPD, the teacher model may provide guidance that discourages the student from moving toward correct trajectories or moves the student toward incorrect ones, which is misaligned with outcome reward. Such misaligned guidance is unreliable, as it would mislead the optimization process and ultimately degrade model performance. To mitigate misaligned teacher guidance, we propose Reward-Aligned On-Policy Distillation (RA-OPD). The key insight is to keep only trajectories whose induced updates move the student toward correct trajectories or discourage the student from moving toward incorrect ones. Specifically, for each sampled trajectory, RA-OPD checks whether its trajectory-level distillation return is consistent with its outcome reward and then filters out the misaligned trajectories. RA-OPD selects more reliable trajectories to improve student model performance without requiring additional computational cost. We evaluate RA-OPD on math and code benchmarks using models from the Qwen3 family and the DeepSeek-R1 family. Across seven math benchmarks and three code benchmarks, RA-OPD significantly outperforms standard OPD and other tested OPD variants.