Sub-Band Knowledge Distillation Framework for Speech Enhancement
Proposes sub-band knowledge distillation, using expert sub-models to improve speech enhancement, outperforming full-band models with fewer parameters.
Key Findings
Methodology
The framework divides the spectral magnitude into multiple sub-bands, training dedicated teacher models for each. These teachers guide a unified student model via distillation loss functions (e.g., L2 and soft-label loss). The architecture employs bidirectional LSTM networks with specific sub-band widths (e.g., 40 frequency points). During training, sub-bands are randomly sampled for the student, which learns to enhance all bands simultaneously without increasing parameters. Experiments on Voice Bank + DEMAND dataset validate that teacher-guided student models outperform full-band counterparts, with performance gains of approximately 0.2 PESQ points and 1.5% STOI, while reducing parameter count by 7%. The approach effectively balances local feature learning and global information integration.
Key Results
- On the Voice Bank + DEMAND dataset, the student model with teacher guidance achieved PESQ scores of 2.5, surpassing the full-band model’s 2.3, and STOI scores of 93.9%, exceeding the baseline’s 92.4%. Parameter reduction was about 7%. Different sub-band widths (20, 40, 80) showed that 40 frequency points offered the best trade-off between performance and complexity. Teacher models trained with larger memory cells (e.g., 512) yielded better guidance, further improving student performance.
- Comparison across model scales indicated that teacher models trained for specific sub-bands outperformed the general student, demonstrating the importance of local expert knowledge. Incorporating teacher guidance consistently improved PESQ and STOI metrics, confirming the effectiveness of the distillation process. The results highlight that local sub-models can capture detailed spectral features, which, when transferred, enhance the overall speech quality.
- The experimental results validate that the proposed sub-band knowledge distillation framework effectively leverages local spectral information, reduces model complexity, and surpasses traditional full-spectrum models, making it highly suitable for resource-constrained deployment scenarios.
Significance
This work advances speech enhancement by integrating local sub-band expertise with global modeling via knowledge distillation, addressing the limitations of full-spectrum models that often require large parameters and computational resources. The method offers a scalable, efficient solution that maintains or improves performance, facilitating deployment in real-world applications such as mobile devices and embedded systems. It also opens new avenues for multi-task learning and multi-model collaboration in speech processing, providing a robust framework for future research in local-global spectral modeling. The approach’s ability to outperform full-band models with fewer parameters marks a significant step toward practical, high-performance speech enhancement solutions.
Limitations
- The sub-band division may lead to loss of cross-band contextual information, especially affecting high-frequency regions where speech energy is sparse, potentially limiting performance in certain noisy conditions.
- Training multiple teacher models increases initial computational cost and complexity, which may hinder scalability for very fine sub-band divisions.
- Generalization across diverse datasets and unseen noise types remains a challenge; further robustness testing and domain adaptation strategies are needed for broader applicability.
Future Work
Future research will explore adaptive multi-scale sub-band partitioning, incorporating attention mechanisms to enhance cross-band feature fusion. End-to-end training frameworks combining spectral and perceptual losses are also planned to optimize both objective and subjective quality. Additionally, integrating multi-task learning for speech recognition and enhancement could improve robustness and real-world performance, especially in highly variable acoustic environments.
AI Executive Summary
Single-channel speech enhancement has traditionally relied on full-spectrum spectral features, demanding high computational resources and large models. While effective, these methods often struggle with deployment in resource-limited environments. Recent advances have shown that local spectral features contain rich information vital for denoising, but leveraging this efficiently remains challenging.
This paper introduces a novel sub-band knowledge distillation framework that addresses these issues by dividing the spectral magnitude into multiple sub-bands. For each sub-band, an expert teacher model is trained to optimize local feature mapping. These teacher models then guide a unified student model through a distillation process, enabling it to learn all sub-bands collectively without increasing parameters. The core mechanism involves a combination of local loss functions and soft-label guidance, implemented via bidirectional LSTM networks.
Experimental validation on the Voice Bank + DEMAND dataset demonstrates that the student model, guided by multiple expert teachers, surpasses the performance of traditional full-spectrum models. Specifically, the PESQ score improves from 2.3 to 2.5, and STOI increases from 92.4% to 93.9%, while reducing parameter count by approximately 7%. The results confirm that carefully designed sub-band division—particularly at 40 frequency points—strikes an optimal balance between complexity and performance.
This approach offers a scalable, efficient solution for real-world speech enhancement applications, especially in scenarios with limited computational resources. It also paves the way for future research into multi-scale spectral modeling, attention-based feature fusion, and multi-task learning strategies, promising further improvements in robustness and generalization.
Despite its success, the method faces limitations such as potential loss of cross-band contextual information and increased initial training complexity due to multiple teacher models. Future work will focus on adaptive sub-band strategies, end-to-end training, and broader robustness testing, aiming to realize more versatile and deployable speech enhancement systems.
Deep Dive
Abstract
In single-channel speech enhancement, methods based on full-band spectral features have been widely studied. However, only a few methods pay attention to non-full-band spectral features. In this paper, we explore a knowledge distillation framework based on sub-band spectral mapping for single-channel speech enhancement. Specifically, we divide the full frequency band into multiple sub-bands and pre-train an elite-level sub-band enhancement model (teacher model) for each sub-band. These teacher models are dedicated to processing their own sub-bands. Next, under the teacher models' guidance, we train a general sub-band enhancement model (student model) that works for all sub-bands. Without increasing the number of model parameters and computational complexity, the student model's performance is further improved. To evaluate our proposed method, we conducted a large number of experiments on an open-source data set. The final experimental results show that the guidance from the elite-level teacher models dramatically improves the student model's performance, which exceeds the full-band model by employing fewer parameters.