On-policy Distillation with Verifiable Reward
OPDVR combines OPD and RLVR, achieving superior performance on six reasoning benchmarks.
Key Findings
Methodology
OPDVR seamlessly combines OPD and RLVR using a ReLU gating mechanism. It first reformulates the implicit reward of sampled-token OPD based on trajectory correctness, then applies a ReLU gating mechanism to ensure correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards, aligning the distillation signal with task success while preserving the teacher's distributional guidance. This modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO.
Key Results
- On six reasoning benchmarks, OPDVR achieves an average accuracy of 49.1, surpassing standard OPD's 47.8.
- On the AIME24 benchmark, OPDVR achieves an accuracy of 36.9, surpassing the teacher model's 36.0.
- In cross-architecture settings, OPDVR scores 30.3 on the AMC benchmark, higher than RLSD's 28.7.
Significance
The OPDVR method holds significant implications for both academia and industry. It addresses the limitations of RLVR and OPD by combining their strengths, providing reliable task-level correctness supervision and dense token-level distributional guidance. This approach not only enhances performance in reasoning tasks but also offers new directions for future research.
Technical Contribution
OPDVR's technical contribution lies in its seamless integration of OPD and RLVR, eliminating the need for additional hyperparameters required by existing methods that rely on weighted combinations or heuristic switching. By employing a ReLU gating mechanism, OPDVR transforms sampled-token OPD into a proper RLVR method, supporting seamless integration with existing RL algorithms like GRPO.
Novelty
OPDVR is the first to apply a ReLU gating mechanism to combine OPD and RLVR, addressing the misalignment of reward signs with trajectory correctness in sampled-token OPD. Unlike existing methods, OPDVR requires no additional hyperparameters, simplifying model training.
Limitations
- OPDVR may still face sparse reward issues in complex tasks, affecting training outcomes.
- Performance may degrade when transferring across different architectures.
Future Work
Future research directions include exploring OPDVR's performance on larger datasets and its integration with other RL algorithms. Further optimization of the ReLU gating mechanism to enhance model generalization is also a key area of interest.
AI Executive Summary
In the field of reinforcement learning, RLVR and OPD are two commonly used post-training paradigms. RLVR provides sparse, task-level reward signals, while OPD offers dense token-level guidance but ignores trajectory correctness. Existing methods often rely on weighted combinations or heuristic switching, introducing additional hyperparameters and trade-offs.
OPDVR seamlessly combines OPD and RLVR using a ReLU gating mechanism, eliminating the need for extra hyperparameters. The method first reformulates the implicit reward of sampled-token OPD based on trajectory correctness, then applies a ReLU gating mechanism to ensure correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards, aligning the distillation signal with task success while preserving the teacher's distributional guidance.
Experimental results show that OPDVR consistently outperforms standard OPD across six reasoning benchmarks, with average accuracy surpassing even the teacher model on certain benchmarks. This demonstrates that OPDVR not only enhances model performance but also offers new directions for future research.
Deep Analysis
Background
In recent years, reinforcement learning has made significant progress in solving complex reasoning tasks. RLVR and OPD are two commonly used post-training paradigms. RLVR optimizes models using task-level reward signals, but sparse rewards lead to insufficient supervision for intermediate steps. OPD provides dense token-level guidance from a teacher model but ignores whether the generated response is correct, limiting the student's performance.
Core Problem
The limitations of RLVR and OPD restrict model performance. RLVR's sparse rewards lead to credit assignment challenges, while OPD ignores trajectory correctness, limiting student performance. Combining their strengths to provide reliable task-level correctness supervision and dense token-level distributional guidance is a pressing issue.
Innovation
OPDVR seamlessly combines OPD and RLVR using a ReLU gating mechanism. It first reformulates the implicit reward of sampled-token OPD based on trajectory correctness, then applies a ReLU gating mechanism to ensure correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards, aligning the distillation signal with task success.
Methodology
- �� Reformulate the implicit reward of sampled-token OPD based on trajectory correctness.
- �� Apply a ReLU gating mechanism to ensure correct trajectories receive non-negative rewards.
- �� Transform sampled-token OPD into a proper RLVR method, supporting integration with any policy gradient algorithm.
Experiments
Experiments were conducted on six reasoning benchmarks, including AIME24, AIME25, AMC, MATH500, Minerva, and OlympiadBench. The Qwen3-4B and Qwen3-1.7B models were used for same-architecture and cross-architecture distillation training, evaluating the performance differences between OPDVR and other baseline methods.
Results
OPDVR consistently outperforms all baselines on six reasoning benchmarks, achieving an average accuracy of 49.1, surpassing standard OPD's 47.8. On the AIME24 benchmark, OPDVR achieves an accuracy of 36.9, surpassing the teacher model's 36.0. In cross-architecture settings, OPDVR scores 30.3 on the AMC benchmark, higher than RLSD's 28.7.
Applications
OPDVR can be applied to scenarios requiring high-precision reasoning, such as mathematical problem solving and code generation. Its combination of OPD and RLVR provides reliable task-level correctness supervision and dense token-level distributional guidance.
Limitations & Outlook
OPDVR may still face sparse reward issues in complex tasks, affecting training outcomes. Performance may degrade when transferring across different architectures. Further exploration of the ReLU gating mechanism's optimization space is needed.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. RLVR is like judging your cooking based on the final taste, but without guidance during the process. OPD is like having a chef guiding you every step, but not caring about the final taste. OPDVR combines the best of both, like having a chef who guides you every step and also gives feedback based on the final taste, ensuring your dish is both beautiful and delicious.
ELI14 Explained like you're 14
Imagine you're playing a game. RLVR is like only getting points when you finish a level, while OPD is like having hints every step but not caring if you finish. OPDVR combines the best of both, giving you hints every step and extra rewards when you finish, so you can finish faster and better!
Glossary
ReLU Gating Mechanism
A mechanism to ensure reward signs align with trajectory correctness.
Used in OPDVR to adjust the reward signs of sampled-token OPD.
OPD
A distillation method providing dense token-level guidance from a teacher model.
Used in the paper to provide distributional guidance for the student model.
RLVR
A method optimizing models using task-level reward signals.
Used in the paper to provide task-level correctness supervision.
GRPO
A policy optimization method for stabilizing reward signals.
Used in the paper in conjunction with OPDVR.
Sampled-Token OPD
A variant of OPD approximating reverse KL gradient using a single token sample.
Used in the paper for integration with RLVR.
Open Questions Unanswered questions from this research
- 1 How to optimize OPDVR's performance on larger datasets?
- 2 How to further improve OPDVR's transfer performance across different architectures?
Applications
Immediate Applications
Mathematical Problem Solving
OPDVR can be used to improve the accuracy of mathematical problem solving, especially in complex reasoning tasks.
Long-term Vision
General Artificial Intelligence
OPDVR's combination method provides possibilities for achieving more efficient general artificial intelligence.
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.