Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Study finds SDPO accelerates in-domain learning under specific conditions but struggles in cross-domain scenarios.
Key Findings
Methodology
This study employs Self-Distillation Policy Optimization (SDPO), combining stable teacher signals and dense token-level supervision to accelerate in-domain specialization. By comparing GRPO and SDPO, the study investigates performance in continual post-training.
Key Results
- SDPO excels in in-domain tasks, improving AIME accuracy to 56.42% in MATH tasks.
- However, SDPO underperforms in cross-domain tasks, especially in intermediate-distance interference tasks like GPQA.
- Experiments show severe forgetting in multi-domain training with SDPO, particularly when teacher signals are unstable.
Significance
This research reveals that dense self-distillation does not automatically inherit the retention benefits of reinforcement learning in continual learning. It emphasizes the importance of teacher signal stability and token-level supervision reliability for model performance.
Technical Contribution
The paper distinguishes two key factors in continual post-training: on-policy data and the objective used to update the model. By comparing SDPO and GRPO, it highlights the limitations and potential risks of dense supervision.
Novelty
This is the first systematic analysis of the limitations of dense self-distillation in continual post-training, particularly its poor performance in cross-domain tasks.
Limitations
- SDPO exhibits significant forgetting in cross-domain tasks, especially in intermediate-distance tasks.
- Unstable teacher signals can lead to model collapse.
Future Work
Future research should focus on enhancing SDPO's cross-domain generalization without increasing forgetting risk and optimizing teacher signal stability.
AI Executive Summary
Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. However, existing research suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation seen as an attractive approach.
This paper revisits this optimistic view through Self-Distillation Policy Optimization (SDPO). Experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well-aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods like GRPO adapt more conservatively and better preserve prior capabilities.
Further analyses reveal that dense self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher-student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training.
Deep Analysis
Background
With the widespread application of large language models, continual learning has become an important research direction. Existing methods often rely on on-policy reinforcement learning to mitigate forgetting, but on-policy self-distillation has gained attention for reducing dependence on external teachers or explicit reward models.
Core Problem
The core problem is how to effectively acquire new knowledge in continual learning without forgetting old knowledge. Especially in multi-domain scenarios, models need to balance between different tasks, and existing methods lack cross-domain generalization capabilities.
Innovation
The innovation of this paper lies in systematically analyzing the limitations of on-policy self-distillation in continual post-training, particularly its poor performance in cross-domain tasks. By comparing SDPO and GRPO, it reveals the potential risks of dense supervision.
Methodology
- �� Use Self-Distillation Policy Optimization (SDPO), combining stable teacher signals and dense token-level supervision.
- �� Compare GRPO and SDPO to study performance in continual post-training.
- �� Analyze drift in parameter space and response space, and the amplification of high-frequency formatting artifacts.
Experiments
The experimental design includes post-training in both single-domain and multi-domain continual settings, comparing GRPO, standard SDPO, and SDPO variants with supervision density. Evaluation is conducted on in-domain and general benchmarks to assess specialization, retention, and transfer capabilities.
Results
Results show that SDPO significantly improves performance on the current training domain but also increases the risk of drift, interference, and even collapse. Supervision density introduces a trade-off: it strengthens the local learning signal but also increases sensitivity, domain mismatch, and accumulated artifacts.
Applications
The findings have significant implications for the design of continual learning models, especially in applications requiring cross-domain generalization. They can be used to optimize post-training strategies for large language models.
Limitations & Outlook
SDPO exhibits significant forgetting in cross-domain tasks, especially in intermediate-distance tasks. Unstable teacher signals can lead to model collapse. Future research should focus on enhancing SDPO's cross-domain generalization without increasing forgetting risk.
Plain Language Accessible to non-experts
Imagine a factory where workers learn new production techniques every day but cannot forget old methods. SDPO is like a new training method that helps workers quickly master new techniques but sometimes causes them to forget old skills, especially when the new and old techniques are similar but not identical. GRPO is more like a conservative training method, slower in learning but better at retaining old skills.
ELI14 Explained like you're 14
Imagine you're playing a game and have learned a lot of tricks for different levels. Now there's a new level, and you want to master it quickly without forgetting the old tricks. SDPO is like a super coach that helps you quickly master the new level but sometimes makes you forget the old tricks, especially when the new level is a bit similar but not exactly the same as the previous ones. GRPO is like a cautious coach, slower in learning but ensures you don't forget the old tricks.
Glossary
Self-Distillation Policy Optimization (SDPO)
A policy optimization method using the model itself as a teacher, providing dense token-level supervision to accelerate learning.
Used to accelerate in-domain specialization but underperforms in cross-domain tasks.
Group Relative Policy Optimization (GRPO)
A policy optimization method that stabilizes reinforcement learning using relative rewards.
Compared with SDPO to study performance in continual post-training.
Continual Post-Training
A training method that enables foundation models to acquire new knowledge while preserving existing capabilities.
Studying how to acquire new knowledge without forgetting old ones.
Dense Supervision
A method to improve learning efficiency through dense token-level supervision.
Used in SDPO to accelerate in-domain specialization.
Cross-Domain Generalization
The ability of a model to balance between different tasks.
Studying SDPO's poor performance in cross-domain tasks.
Open Questions Unanswered questions from this research
- 1 How to enhance SDPO's cross-domain generalization without increasing forgetting risk.
- 2 How to optimize teacher signal stability to improve SDPO performance.
Applications
Immediate Applications
Continual Learning Model Optimization
Can be used to optimize post-training strategies for large language models, especially in applications requiring cross-domain generalization.
Long-term Vision
Adaptive Learning in Intelligent Systems
In the future, it can be applied to adaptive learning in intelligent systems, allowing them to better balance new and old knowledge across different tasks.
Abstract
Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.