Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
Unified on-policy self-distillation (USD) dynamically adjusts token weights and privileged information via a single dual variable, boosting reasoning performance.
Yongkang Yang, Zhezheng Hao, Hong Zhang et al.