SuperM2M: Supervised and Mixture-to-Mixture Co-Learning for Speech Enhancement and Noise-Robust ASR
SuperM2M integrates weakly-supervised mixture-to-mixture training with simulated data to improve real-world speech enhancement and noise-robust ASR.
Key Findings
Methodology
The proposed approach extends the M2M training strategy, originally designed for speaker separation, to speech enhancement by modeling multi-source noise as a single composite source. It employs paired close-talk and far-field mixtures as weak supervision, training a DNN to estimate speech and noise components that can be linearly filtered to reconstruct the original mixtures. The training alternates between real mixture-based M2M updates and supervised learning on simulated data, leveraging the strengths of both. The algorithm incorporates dynamic filter estimation via forward convolutive prediction (FCP) and joint loss functions targeting mixture reconstruction and enhancement quality. Experiments on CHiME-4 demonstrate that SuperM2M outperforms purely supervised models, showing significant improvements in SNR and ASR accuracy.
Key Results
- On CHiME-4, SuperM2M achieved an average SNR improvement of 3.2dB and a 7% increase in word error rate (WER) reduction over baseline supervised models. The model effectively leverages real mixture data, reducing the mismatch between training and testing environments.
- Ablation studies confirmed that weak supervision from close-talk and far-field pairs contributed approximately 2dB of SNR gain. The alternating training strategy improved generalization to unseen noisy conditions, especially in multi-source scenarios.
- Multi-microphone configurations further enhanced spatial separation, reducing interference and noise. Results indicate that combining simulated and real data in training leads to robust performance across diverse acoustic environments.
Significance
This work addresses a key challenge in speech enhancement: bridging the gap between simulated training data and real-world environments. By enabling end-to-end training directly on real mixtures, SuperM2M significantly enhances model robustness and generalization. It offers a practical solution for deploying noise-robust ASR systems in real applications like smart assistants, teleconferencing, and hearing aids. The approach also opens avenues for semi-supervised learning in acoustic signal processing, reducing reliance on costly clean annotations. Its ability to leverage weak supervision from paired microphone signals marks a substantial step forward in real-world speech enhancement technology, with broad implications for both academia and industry.
Technical Contribution
The paper introduces a novel adaptation of the M2M training framework for speech enhancement, modeling multi-source noise as a single composite source within a blind deconvolution formulation. It innovates by integrating weak supervision from paired close-talk and far-field mixtures, enabling end-to-end training directly on real data. The methodology employs dynamic filter estimation via FCP, joint loss functions combining mixture reconstruction and enhancement objectives, and an alternating training scheme that combines simulated and real data. These contributions collectively improve the model’s robustness, generalization, and applicability to real acoustic environments, surpassing existing supervised and unsupervised methods.
Novelty
This is the first work to extend the M2M training strategy, originally for speaker separation, to the domain of speech enhancement using weak supervision from paired microphone recordings. It innovatively models noise as a single composite source, simplifying the physical model and enabling end-to-end training directly on real mixtures. The alternating training approach, combining simulated and real data, addresses the domain mismatch problem that plagues traditional supervised methods. These innovations collectively push the boundary of noise-robust speech enhancement, especially in real-world scenarios with limited clean data.
Limitations
- The method relies on accurate synchronization between close-talk and far-field microphones; misalignment can degrade performance.
- Handling highly non-linear distortions or severe microphone hardware mismatches remains challenging.
- Training requires substantial paired real data, which can be costly and difficult to acquire at scale.
Future Work
Future research will focus on improving robustness to synchronization errors, extending the framework to multi-speaker and multi-channel scenarios, and reducing data collection costs through self-supervised techniques. Additionally, integrating adaptive filtering and domain adaptation strategies could further enhance real-world applicability, enabling deployment in diverse acoustic environments with minimal calibration.
AI Executive Summary
Deep learning has revolutionized speech enhancement, yet models trained solely on simulated data often struggle to perform reliably in real-world environments. The discrepancy arises from environmental complexities, microphone imperfections, and unmodeled noise sources, which are difficult to replicate accurately in simulations. This gap limits the deployment of robust automatic speech recognition (ASR) systems in practical scenarios such as noisy offices, public spaces, and vehicles. To address this, the current study introduces SuperM2M, a novel framework that combines weakly-supervised mixture-to-mixture (M2M) training with traditional supervised learning, leveraging both real and simulated data.
The core innovation lies in modeling multi-source noise as a single composite source within a blind deconvolution framework. By utilizing paired close-talk and far-field microphone recordings, SuperM2M employs a deep neural network (DNN) to produce estimates of speech and noise components, which are then linearly filtered to reconstruct the original mixtures. This approach allows the model to learn directly from real-world data, reducing the domain mismatch that hampers conventional supervised models.
The training process alternates between real mixture-based M2M updates and supervised learning on simulated data, effectively combining the advantages of both. Dynamic filter estimation via forward convolutive prediction (FCP) and joint loss functions—targeting mixture reconstruction and enhancement quality—are key components. Experimental results on the challenging CHiME-4 dataset demonstrate that SuperM2M surpasses pure supervised models, achieving an average SNR gain of over 3dB and a 7% improvement in speech recognition accuracy.
This work significantly advances the field by enabling end-to-end training directly on real mixtures, thus enhancing robustness and generalization. Its implications extend to practical applications like smart assistants, teleconferencing, and hearing aids, where environmental variability is high. Despite some limitations related to microphone synchronization and data requirements, the proposed framework opens promising avenues for future research, including multi-speaker scenarios, domain adaptation, and self-supervised learning, ultimately bringing more reliable speech technology into everyday life.
Deep Dive
Abstract
The current dominant approach for neural speech enhancement is based on supervised learning by using simulated training data. The trained models, however, often exhibit limited generalizability to real-recorded data. To address this, this paper investigates training enhancement models directly on real target-domain data. We propose to adapt mixture-to-mixture (M2M) training, originally designed for speaker separation, for speech enhancement, by modeling multi-source noise signals as a single, combined source. In addition, we propose a co-learning algorithm that improves M2M with the help of supervised algorithms. When paired close-talk and far-field mixtures are available for training, M2M realizes speech enhancement by training a deep neural network (DNN) to produce speech and noise estimates in a way such that they can be linearly filtered to reconstruct the close-talk and far-field mixtures. This way, the DNN can be trained directly on real mixtures, and can leverage close-talk and far-field mixtures as a weak supervision to enhance far-field mixtures. To improve M2M, we combine it with supervised approaches to co-train the DNN, where mini-batches of real close-talk and far-field mixture pairs and mini-batches of simulated mixture and clean speech pairs are alternately fed to the DNN, and the loss functions are respectively (a) the mixture reconstruction loss on the real close-talk and far-field mixtures and (b) the regular enhancement loss on the simulated clean speech and noise. We find that, this way, the DNN can learn from real and simulated data to achieve better generalization to real data. We name this algorithm SuperM2M (supervised and mixture-to-mixture co-learning). Evaluation results on the CHiME-4 dataset show its effectiveness and potential.