Blockwise Policy-Drift Gating for On-Policy Distillation
Introduces blockwise policy-drift gating, improving OPD pass rates in math reasoning tasks.
Key Findings
Methodology
The paper introduces a lightweight student-only old-current drift controller called blockwise policy-drift gating. This method computes log-probability shifts between the behavior student and the current student on the sampled token path, aggregates these shifts over fixed blocks or spans, and uses the resulting mean-normalized gates to reweight OPD position losses. It does not change teacher targets, teacher supports, or the rollout policy.
Key Results
- Fixed 64-token block gating improves sampled-token OPD mean pass@8 from 0.4978 to 0.5160 across AIME24, AIME25, MATH500, and AMC23.
- On Teacher-TopK/LSM, Block64 gives the best four-benchmark mean pass@8 among trained students.
- The results identify local old-current policy drift as a practical control signal.
Significance
This study addresses the instability of OPD in long reasoning tasks by introducing blockwise policy-drift gating. By incorporating drift control signals in the student policy, it enhances the robustness of OPD, particularly in math reasoning tasks. This method provides a new approach to addressing the long-standing fragility of OPD training signals.
Technical Contribution
Technically, the paper proposes a new policy drift control method that weights OPD losses by aggregating log-probability shifts over blocks. Unlike existing OPD methods, it does not rely on changes in teacher signals but adjusts loss weights through student policy drift.
Novelty
This paper is the first to apply blockwise policy drift to OPD, proposing a drift control method independent of teacher signals. Compared to existing OPD methods, it provides a new perspective by adjusting loss weights through student policy drift.
Limitations
- The method's results on AIME25 may be unstable due to small sample sizes.
- It does not test asynchronous execution, sample buffering, or refresh strategies.
- Validated only in math reasoning tasks, applicability to other domains is unknown.
Future Work
Future research could expand to other models and datasets, test different block sizes and drift statistics methods. Additionally, asynchronous training systems and sample refresh strategies could be introduced to enhance applicability and robustness.
AI Executive Summary
In long reasoning tasks, existing OPD methods are prone to instability, especially when student policies deviate from teacher support regions. This paper proposes a new method called blockwise policy-drift gating, which computes log-probability shifts between the behavior student and the current student on the sampled token path, aggregates these shifts over fixed blocks, and uses the resulting mean-normalized gates to reweight OPD position losses.
Experimental results show that fixed 64-token block gating significantly improves OPD mean pass rates, particularly in math reasoning tasks. This method does not rely on changes in teacher signals but adjusts loss weights through student policy drift, providing a new approach to addressing the long-standing fragility of OPD training signals.
While the method performs well in math reasoning tasks, its applicability to other domains remains unverified. Future research could expand to other models and datasets, test different block sizes and drift statistics methods to enhance applicability and robustness.
Deep Analysis
Background
In the field of natural language processing, knowledge distillation is a common method for transferring the behavior of a teacher model to a smaller student model. However, traditional offline distillation methods suffer from a train-test distribution gap. OPD addresses this issue by computing teacher signals on trajectories sampled by the student itself. However, in long reasoning tasks, OPD training signals can become fragile, especially when student policies deviate from teacher support regions.
Core Problem
In long reasoning tasks, OPD is prone to instability, especially when student policies deviate from teacher support regions. Existing methods mainly rely on changes in teacher signals, but these methods are limited when student policies drift. Therefore, a new method is needed to control student policy drift and improve OPD robustness.
Innovation
This paper proposes a new method called blockwise policy-drift gating, which computes log-probability shifts between the behavior student and the current student on the sampled token path, aggregates these shifts over fixed blocks, and uses the resulting mean-normalized gates to reweight OPD position losses. This method does not rely on changes in teacher signals but adjusts loss weights through student policy drift.
Methodology
- �� Compute log-probability shifts between the behavior student and the current student on the sampled token path.
- �� Aggregate these shifts over fixed blocks.
- �� Use the resulting mean-normalized gates to reweight OPD position losses.
- �� Do not change teacher targets, teacher supports, or the rollout policy.
Experiments
Experiments were conducted on the Qwen3 math reasoning benchmark with a fixed 200-step training budget. Pass@8 was used as the main problem-level solve-rate metric. Results show that fixed 64-token block gating significantly improves OPD mean pass rates, especially in math reasoning tasks.
Results
Results show that fixed 64-token block gating significantly improves OPD mean pass rates, especially in math reasoning tasks. This method does not rely on changes in teacher signals but adjusts loss weights through student policy drift, providing a new approach to addressing the long-standing fragility of OPD training signals.
Applications
The method performs well in math reasoning tasks and is suitable for scenarios requiring high robustness and solve rates. It could be extended to other fields such as natural language generation and dialogue systems.
Limitations & Outlook
While the method performs well in math reasoning tasks, its applicability to other domains remains unverified. Future research could expand to other models and datasets, test different block sizes and drift statistics methods to enhance applicability and robustness.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. You have a recipe (teacher signal), but you decide to adjust some steps to your taste (student policy). Sometimes, you might stray from the recipe, leading to a dish that doesn't taste as expected. Blockwise policy-drift gating is like pausing after each step to taste the dish and adjust the seasoning as needed. This method helps you make a delicious dish without fully relying on the recipe.
ELI14 Explained like you're 14
Imagine you're playing a game with a guide (teacher signal), but you like to explore on your own (student policy). Sometimes, you might stray from the guide, leading to game failure. Blockwise policy-drift gating is like pausing after each level to see how far you've gone and adjust your strategy as needed. This method helps you beat the game without fully relying on the guide!
Glossary
Policy Drift
Refers to the phenomenon where the student policy deviates from the teacher signal during training.
Used to measure the deviation of student policy from teacher signals.
Knowledge Distillation
A method for transferring the behavior of a teacher model to a student model.
Used to train a smaller student model to mimic the behavior of a teacher model.
OPD (On-Policy Distillation)
A distillation method that computes teacher signals on trajectories sampled by the student itself.
Used to address the train-test distribution gap.
Blockwise Gating
A method that weights losses by aggregating log-probability shifts over fixed blocks.
Used to adjust OPD loss weights.
Math Reasoning
The process of understanding and solving mathematical problems.
Used to evaluate model performance in math tasks.
Open Questions Unanswered questions from this research
- 1 How to apply blockwise policy-drift gating in other domains?
- 2 How to effectively control policy drift in asynchronous training systems?
Applications
Immediate Applications
Math Reasoning Tasks
Improves solve rates and robustness in math reasoning tasks, applicable in education.
Long-term Vision
Natural Language Generation
Apply in natural language generation tasks to improve text quality and consistency.
Abstract
On-policy distillation (OPD) trains a student policy using teacher signals computed on trajectories sampled by the student itself. Recent work shows that sampled-token OPD can be fragile on long-horizon reasoning tasks and that local teacher-support matching is a simple and effective repair. This paper introduces blockwise policy-drift gating, a lightweight student-only old-current drift controller for OPD under rollout reuse. The method computes log-probability shifts between the behavior student and the current student on the sampled token path, aggregates these shifts over fixed blocks or spans, and uses the resulting detached, mean-normalized gates to reweight OPD position losses. It does not change teacher targets, teacher top-K supports, or the rollout policy. In a six-variant Qwen3 math reasoning benchmark with a uniform 200-step training budget for all trained variants, we use pass@8 as the primary problem-level solve-rate metric. Fixed 64-token block gating improves sampled-token OPD mean pass@8 from 0.4978 to 0.5160 across AIME24, AIME25, MATH500, and AMC23. On Teacher-TopK/LSM, Block64 gives the best four-benchmark mean pass@8 among trained students. The results identify local old-current policy drift as a practical control signal for reused OPD rollouts and motivate block-level gating as a simple default for improving solve-rate robustness.