Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
Introduces teachability metric for token selection in on-policy distillation, improving performance with only 5% tokens.
Key Findings
Methodology
This paper proposes a fixed-context diagnostic measuring teacher-student KL reduction in identical contexts to distinguish support-aligned (learnable) from support-off (incompatible) disagreement. The teachability score combines local support mass and disagreement, enabling lightweight token position filtering without auxiliary models. Experiments on Qwen models demonstrate that selecting high-teachability tokens with only 5% of total tokens often matches or exceeds full distillation performance, emphasizing the importance of local support relations in supervision signal quality.
Key Results
- In Qwen3-4B to 1.7B transfer, TA-OPD with 5% tokens improved average accuracy from 42.37% to 44.89%, outperforming entropy- and divergence-based selectors across multiple benchmarks such as AIME24 and GPQA-Diamond.
- In cross-architecture distillation (DeepSeek-R1 to Qwen2.5), using only 5% tokens, TA-OPD achieved 2-3 point performance gains over full-token distillation, validating the effectiveness of teachability-based filtering in noisy, high-divergence scenarios.
- Support mass (support support) strongly correlates with performance gains, indicating that filtering based on local support relations effectively removes unhelpful supervision signals.
Significance
This work shifts the focus from salience-based token selection to support-aware supervision, enabling more efficient knowledge transfer. It addresses the long-standing challenge of noisy supervision signals in large-scale model distillation, offering a theoretically grounded and practically effective method that reduces supervision costs while maintaining or improving downstream task performance.
Technical Contribution
The paper introduces a formal measure of token teachability based on local support relations and KL reduction, decomposes disagreement into learnable and incompatible components, and designs a lightweight selection algorithm (TA-OPD). This approach leverages probabilistic support support to filter supervision signals, providing theoretical guarantees of improved local learning efficiency and empirical validation across multiple models and tasks.
Novelty
First to formalize local support relations as a measure of token teachability in on-policy distillation, moving beyond traditional salience or uncertainty metrics. The support-aware filtering mechanism offers a new perspective on supervision signal quality, enabling high efficiency with minimal supervision tokens, a significant innovation over prior approaches.
Limitations
- The method relies on probabilistic support estimates, which may be less accurate in highly noisy or out-of-distribution contexts. Its effectiveness in multilingual, multimodal, or dialogue tasks remains to be validated.
- Support relation computation adds some overhead, and the approach does not explicitly model token interactions or global context, which could further enhance performance.
- Current experiments focus on large language models and math reasoning tasks; broader generalization requires further testing.
Future Work
Future directions include extending support relation modeling to multi-modal and multilingual settings, integrating global context awareness, and combining with reinforcement learning or self-supervised signals for adaptive supervision. Exploring support relations in model pruning, transfer learning, and continual learning could further enhance efficiency and robustness.
AI Executive Summary
Large-scale language models have revolutionized NLP but pose significant challenges in computational cost and efficiency. Knowledge distillation offers a promising solution by transferring knowledge from a large teacher to a smaller student, but traditional methods often rely on dense supervision signals that are costly and sometimes noisy. Recent advances introduced token-level selection strategies based on uncertainty or divergence, yet these approaches overlook the fundamental question: are these signals truly learnable?
This paper addresses this gap by proposing a novel measure called teachability, which evaluates whether a teacher's supervision signal at a specific token position can be effectively absorbed by the student. The core idea hinges on a fixed-context diagnostic that measures the reduction in teacher-student KL divergence within identical contexts, thereby distinguishing between support-aligned (learnable) and support-off (incompatible) disagreement. The authors formalize this distinction through local support relations, enabling a lightweight token filtering mechanism that prioritizes high-teachability positions.
Extensive experiments across multiple models (Qwen3, Qwen2.5, DeepSeek-R1) and diverse tasks demonstrate that selecting only 5% of tokens based on teachability often matches or surpasses full-token distillation. Notably, in scenarios with high divergence and noise, teachability-based filtering significantly improves downstream performance, validating the importance of local support relations in supervision quality.
This work shifts the paradigm from salience-driven token selection to support-aware supervision, offering both theoretical insights and practical algorithms that enhance distillation efficiency. The findings have broad implications for model compression, transfer learning, and resource-constrained deployment, paving the way for more intelligent and effective knowledge transfer methods.
Despite its success, the approach has limitations, including reliance on probabilistic support estimates and focus on language tasks. Future research could extend support modeling to multi-modal data, incorporate global context, and explore integration with reinforcement learning. Overall, this study provides a significant step toward understanding and leveraging the true learnability of teacher signals in large-scale model distillation.
Deep Dive
Abstract
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually learnable? Using a fixed-context diagnostic that measures same-context teacher-student KL reduction, we show that raw KL disagreement is a coarse proxy for learning value. It conflates learnable disagreement, where the teacher assigns corrective mass to the student's top-K candidates, with incompatible disagreement, where the teacher places mass mostly off the student's current support. We formalize this local compatibility as token teachability and show that it better predicts fixed-context improvement than raw KL alone. Motivated by this finding, we propose Teachability-Aware OPD (TA-OPD), a lightweight token-position selection method that applies OPD loss to high-teachability positions without reward models or verifiers. Across Qwen2.5 and Qwen 3 teacher-student settings, TA-OPD often surpasses full-token OPD with only 5% retained tokens and improves over entropy- and divergence-based baselines. Our results reframe selective OPD as selecting learnable teacher signals rather than merely salient tokens.
References (20)
TIP: Token Importance in On-Policy Distillation
Yuanda Xu, Hejian Sang, Zhengze Zhou et al.
InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion
Yuanyi Wang, Zhao-Yi Yan, Yiming Zhang et al.
Model Merging Scaling Laws in Large Language Models
Yuanyi Wang, Yanggan Gu, Yiming Zhang et al.
LLM-Oriented Token-Adaptive Knowledge Distillation
Xurong Xie, Zhucun Xue, Jiafu Wu et al.
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, O. Vinyals, J. Dean
Measuring Mathematical Problem Solving With the MATH Dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath et al.
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou et al.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland et al.
MiniPLM: Knowledge Distillation for Pre-Training Language Models
Yu-Xian Gu, Hao Zhou, Fandong Meng et al.
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
Songming Zhang, Xue Zhang, Tong Zhang et al.
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
Shenzhi Wang, Le Yu, Chang Gao et al.
Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruning
Yi-Ju Guo, Wenkai Yang, Zexu Sun et al.
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
Wenjun Wang, Yanggan Gu, Shuo Cai et al.
MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging
Yuanyi Wang, Yanggan Gu, Zihao Wang et al.
On-Policy Context Distillation for Language Models
Tianzhu Ye, Li Dong, Xun Wu et al.
Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
Wenkai Yang, Weijie Liu, Ruobing Xie et al.
Entropy-Aware On-Policy Distillation of Language Models
Woogyeol Jin, Taywon Min, Yongjin Yang et al.
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Yaxuan Li, Yu-Xin Zuo, Bingxiang He et al.
OPSDL: On-Policy Self-Distillation for Long-Context Language Models
Xin-Sen Zhang, Zhe Ding, Tian Pan et al.
FeatCal: Feature Calibration for Post-Merging Models
Yanggan Gu, Shuo Cai, Zihao Wang et al.
Cited By (19)
When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation
Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
DOPD: Dual On-policy Distillation
TREK: Distill to Explore, Reinforce to Refine
From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
Mismatch Matters: On-Policy Distillation Beyond Token Agreement
CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection