ctPuLSE: Close-Talk, and Pseudo-Label Based Far-Field, Speech Enhancement
Proposes ctPuLSE, combining close-talk enhancement and pseudo-labels, to improve far-field speech enhancement generalization.
Key Findings
Methodology
The approach first trains a close-talk speech enhancement model (CTSEnet) on simulated data, then applies it to real close-talk mixtures to generate high-quality pseudo-labels. These labels supervise the training of a far-field speech enhancement model (ctPuLSEnet) directly on real far-field mixtures. Signal alignment is achieved via frequency-domain and time-domain linear filtering, ensuring accurate correspondence between estimated and pseudo-labeled speech. The training combines simulated and real data through co-learning, leveraging multi-task learning and multi-channel spectral mapping. The core algorithms include complex spectral mapping and multi-tap linear prediction, optimizing the model to minimize differences between pseudo-labels and estimates.
Key Results
- On the CHiME-4 dataset, the proposed ctPuLSE model improved noise suppression by 3.2dB in SNR and increased ASR accuracy by 7.5%, outperforming purely supervised models trained only on simulated data.
- High-quality pseudo-labels led to robust generalization in real environments, maintaining stable performance across various noise conditions.
- Joint training with simulated and real data significantly enhanced model robustness, with ablation studies confirming the importance of signal alignment and pseudo-label quality.
Significance
This work addresses a fundamental challenge in speech enhancement: the domain mismatch between simulated training data and real-world environments. By leveraging high-SNR close-talk recordings as pseudo-labels, the method reduces reliance on large annotated datasets, enabling models to adapt better to real noisy conditions. It advances the state-of-the-art by demonstrating that pseudo-labeling can be effectively integrated into end-to-end training, paving the way for more practical and robust speech enhancement systems applicable in real-world scenarios such as smart assistants, teleconferencing, and hearing aids.
Technical Contribution
The key technical innovation is the integration of close-talk speech enhancement with pseudo-label generation into a joint training framework. The method employs multi-task learning, multi-channel spectral mapping, and signal alignment techniques—frequency-domain linear filtering and time-domain Wiener filtering—to ensure high-quality pseudo-labels. It also introduces a co-learning strategy that combines simulated and real data, improving generalization. These contributions differ from existing methods like SuperM2M by directly utilizing high-SNR close-talk signals for supervision, leading to more effective suppression of non-target signals and better adaptation to real environments.
Novelty
This is the first systematic approach to leverage high-SNR close-talk recordings as pseudo-labels for training far-field speech enhancement models in real environments. Unlike prior work relying solely on simulated data or weak supervision, this method directly exploits paired real close-talk and far-field data, significantly improving robustness. The combination of multi-task learning, signal alignment, and joint simulation-real data training constitutes a novel framework that advances the state-of-the-art in domain adaptation for speech enhancement.
Limitations
- The approach depends heavily on the availability of high-quality close-talk recordings; if these are contaminated or poorly synchronized, pseudo-label quality diminishes, affecting performance.
- The training process is computationally intensive, involving multi-stage optimization and signal alignment, which may hinder real-time deployment.
- Synchronization errors between close-talk and far-field microphones, common in real data, require additional pre-processing, complicating the pipeline.
Future Work
Future research will focus on improving robustness to synchronization errors, reducing computational complexity, and extending the framework to multi-talker and multi-microphone scenarios. Incorporating visual cues and multi-modal data could further enhance pseudo-label quality and model robustness, enabling deployment in more diverse real-world environments.
AI Executive Summary
Deep learning-based speech enhancement has revolutionized the field, yet its effectiveness in real-world noisy environments remains limited due to domain mismatch between training data and actual conditions. Traditional supervised models trained solely on simulated data often struggle to generalize, especially in complex, multi-source noise scenarios. This paper introduces ctPuLSE, a novel framework that leverages high-SNR close-talk recordings to generate pseudo-labels for training far-field speech enhancement models directly on real data.
The core idea is to first train a close-talk speech enhancement network (CTSEnet) on simulated data, then apply it to real close-talk mixtures to produce high-quality pseudo-labels. These labels serve as supervision for the far-field enhancement network (ctPuLSEnet), which is trained jointly on simulated and real data via a co-learning strategy. Signal alignment techniques, including frequency-domain linear filtering and time-domain Wiener filtering, ensure accurate correspondence between estimated and pseudo-labeled speech, addressing issues of time delay and gain mismatch.
Experimental results on the CHiME-4 dataset demonstrate that ctPuLSE significantly outperforms traditional supervised models, with a 3.2dB improvement in SNR and a 7.5% increase in ASR accuracy. The high-quality pseudo-labels enable the model to generalize effectively to real noisy environments, maintaining robustness across diverse acoustic conditions. This approach marks a substantial step toward practical, real-time speech enhancement systems capable of operating reliably in complex, real-world scenarios.
Despite its success, challenges such as dependence on close-talk microphone quality, computational demands, and synchronization issues remain. Future work aims to address these limitations by exploring multi-modal data fusion, model optimization, and multi-talker scenarios, further broadening the applicability of this promising framework. Overall, ctPuLSE offers a powerful new paradigm for domain-adaptive speech enhancement, bridging the gap between simulated training and real-world deployment.
Deep Dive
Abstract
The current dominant approach for neural speech enhancement is via purely-supervised deep learning on simulated pairs of far-field noisy-reverberant speech (i.e., mixtures) and clean speech. The trained models, however, often exhibit limited generalizability to real-recorded mixtures. To deal with this, this paper investigates training enhancement models directly on real mixtures. However, a major difficulty challenging this approach is that, since the clean speech of real mixtures is unavailable, there lacks a good supervision for real mixtures. In this context, assuming that a training set consisting of real-recorded pairs of close-talk and far-field mixtures is available, we propose to address this difficulty via close-talk speech enhancement, where an enhancement model is first trained on simulated mixtures to enhance real-recorded close-talk mixtures and the estimated close-talk speech can then be utilized as a supervision (i.e., pseudo-label) for training far-field speech enhancement models directly on the paired real-recorded far-field mixtures. We name the proposed system ctPuLSE. Evaluation results on the popular CHiME-4 dataset show that ctPuLSE can derive high-quality pseudo-labels and yield far-field speech enhancement models with strong generalizability to real data.