When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Proposes two reliability-aware distillation methods, CHAD and EWAD+CPDP, outperforming standard KD with +0.0219 ROUGE-L on low-resource summarization.
Key Findings
Methodology
Using BanSum Bangla dataset, the study employs gradient alignment to assess per-sample KD usefulness, training a lightweight gate for sample-wise weighting. EWAD combines token-level confidence gating based on teacher maximum probability and introduces a second incompatible teacher with geometric constraints to guide the student. Both methods significantly outperform uniform KD, with only 60M parameters, surpassing a 50× larger fine-tuned Qwen-2.5-3B model.
Key Results
- On BanSum, CHAD achieves ROUGE-L of 0.2541, EWAD+CPDP reaches 0.2587, both outperforming standard KD (0.2368) and CE baseline (0.2365). The improvements are +0.0173 and +0.0219 respectively, with parameter efficiency.
- Across 15 diverse languages, EWAD+CPDP improves ROUGE-L in 10/15 cases, especially where teachers provide complementary signals, demonstrating robustness across typologies.
- Compared to large models like Qwen-2.5-3B, the proposed methods with only 60M parameters outperform them, indicating effective knowledge transfer and model compression.
Significance
This work addresses the core challenge of reliable knowledge transfer in low-resource settings, emphasizing the importance of sample-level trustworthiness over naive confidence metrics. By introducing gradient alignment and geometric constraints, it enables more precise selection of useful training signals, leading to substantial performance gains with smaller models. These advances have broad implications for deploying efficient NLP systems in resource-constrained environments, especially for underrepresented languages, and open pathways for future research in adaptive distillation and multi-source knowledge fusion.
Technical Contribution
The paper's key technical innovations include the gradient alignment-based scoring (CHAD), which directly measures the impact of each sample on validation loss, and the cross-vocabulary geometric constraint (CPDP), which leverages structural relationships between incompatible teachers. These mechanisms enable dynamic, data-driven weighting of distillation signals, moving beyond heuristic confidence proxies. The integration of these techniques into a unified framework demonstrates a significant leap in selective distillation, with theoretical grounding and practical efficacy validated through extensive experiments.
Novelty
This is the first systematic application of gradient-based, sample-level signals for reliability-aware distillation in sequence-to-sequence tasks. Unlike prior methods relying on teacher confidence or surface metrics, the proposed gradient alignment provides a direct measure of sample utility. The introduction of a geometric constraint for incompatible teachers further distinguishes this work, enabling multi-source knowledge fusion without vocabulary alignment, a novel approach in model compression.
Limitations
- The effectiveness of gradient alignment depends on the calibration of the validation gradient; in cases of poor calibration or distribution shift, the signal may be less reliable.
- Computational overhead increases due to gradient calculations, which may limit scalability in very large datasets or models.
- Experiments are primarily focused on summarization; applicability to other generation tasks remains to be validated.
Future Work
Future research should explore more efficient methods for estimating reliability signals, extend the approach to multi-task and multi-modal settings, and investigate automatic calibration techniques. Additionally, integrating these strategies into end-to-end training pipelines and exploring their impact on other NLP tasks like translation or question answering could broaden their utility.
AI Executive Summary
In the realm of natural language processing, model compression and low-resource language applications have long faced the challenge of balancing performance and efficiency. Knowledge distillation (KD) has emerged as a key technique, enabling smaller models to learn from larger teachers. However, traditional KD treats all training samples equally, assuming the teacher’s soft labels are uniformly reliable. This assumption often fails in practice, especially in low-resource settings where teacher models may be poorly calibrated or contain noisy signals.
This paper addresses this critical gap by proposing two novel, reliability-aware distillation methods: CHAD (Counterfactual Harm-Aware Distillation) and EWAD+CPDP (Entropy-Weighted Adaptive Distillation with Cross-Teacher Geometric Constraint). CHAD leverages the concept of gradient alignment, comparing the gradient of each sample’s KD loss with the validation loss gradient to assess its usefulness. A lightweight gate is trained to predict this alignment score, dynamically modulating the sample’s contribution to the overall distillation process. EWAD, on the other hand, operates at the token level, gating each token’s distillation weight based on the teacher’s maximum probability, and incorporates a second, incompatible teacher through a geometric distance constraint to guide the student’s structure.
Extensive experiments on the BanSum Bangla dataset demonstrate that both methods significantly outperform standard KD, with improvements of +0.0173 and +0.0219 ROUGE-L, respectively. Remarkably, these smaller models (60M parameters) surpass a large fine-tuned 2.5-3B parameter model, Qwen-2.5-3B, highlighting the efficiency of reliability-based sample selection. Further tests across 15 diverse languages reveal that the benefits are most pronounced when teachers provide complementary signals, emphasizing the importance of model diversity.
The findings challenge the conventional wisdom that more teacher signal always yields better results. Instead, they underscore that selectively trusting the teacher based on measured reliability can unlock substantial gains. This work paves the way for more intelligent, resource-efficient NLP systems, especially vital for underrepresented languages and low-resource scenarios. Future directions include reducing the computational overhead of gradient-based signals, extending to other tasks, and integrating multi-source knowledge more seamlessly, promising a new paradigm in model distillation and transfer learning.
Deep Dive
Abstract
Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.0003 over a cross-entropy baseline, and that approximately 51.3% of training samples are estimated to actively harm student validation loss under standard KD. We propose two complementary reliability-aware distillation methods. CHAD (Counterfactual Harm-Aware Distillation) measures per-sample KD usefulness via gradient alignment with the validation loss direction and trains a lightweight gate that generalizes this counterfactual judgment to the full training set. EWAD+CPDP combines token-level entropy-weighted adaptive distillation with a capacity-proportional geometric constraint from a second, vocabulary-incompatible teacher. On BanSum, both methods substantially outperform standard KD: CHAD by +0.0173 ROUGE-L and EWAD+CPDP by +0.0219 ROUGE-L, where standard KD itself improves ROUGE-L by only +0.0003; despite using only 60M parameters, both outperform a fine-tuned Qwen 2.5-3B model (50x larger). We further evaluate the stronger method, EWAD+CPDP, across 15 typologically diverse XL-Sum languages organised into three sets, beating the CE-only baseline on 10/15 languages; gains are most reliable where the two teachers contribute complementary signal, and weakest where they have saturated or jointly weak target-language coverage. We release code and trained models to support reproducibility and further research on selective distillation.