On the Importance of Neural Wiener Filter for Resource Efficient Multichannel Speech Enhancement
Proposes a low-latency, resource-efficient multichannel speech enhancement framework using joint training of two lightweight DNNs and a neural Wiener filter, achieving 2ms latency.
Key Findings
Methodology
This paper introduces a time-domain multichannel speech enhancement framework composed of two lightweight recurrent neural networks (LLRNNs) and a neural Wiener filter (NWF). DNN1 enhances noisy signals to estimate NWF coefficients, while DNN2 refines the NWF output. The NWF, similar to traditional frequency-domain Wiener filters, is trained jointly with analysis and synthesis transforms to optimize low-latency performance, reducing nonlinear distortions. The training employs end-to-end joint optimization, outperforming sequential training. The system achieves an algorithmic latency of just 2 milliseconds, with fewer parameters and lower computational complexity. The process involves: • Extracting features via STFT; • Training analysis transforms for filter coefficient estimation; • Using Woodbury formula for matrix inversion; • End-to-end joint training of all components.
Key Results
- On the Interspeech 2020 DNS dataset, the proposed method surpasses existing models with STOI reaching 89.1%, PESQ scores improving from 1.63 to 2.70 (a 65% increase), and SI-SDR improving from -7.48dB to 7.0dB, demonstrating excellent noise suppression and speech fidelity. The model parameters are reduced to 56.63% of comparable models, with approximately half the computational load. The joint training strategy significantly boosts performance over sequential training, validating the effectiveness of the end-to-end approach.
- Compared to baseline models like MC-Conv-TasNet and UXNet, the proposed framework achieves performance gains of over 1.8% in STOI while maintaining low latency. Ablation studies show that optimizing analysis and synthesis transforms is crucial, and joint training yields the best results, confirming the synergy among components.
- Experimental results confirm that the trainable NWF with jointly optimized transforms effectively reduces nonlinear distortions, matching or exceeding DNN-only performance with fewer parameters. The approach demonstrates robustness across various noise conditions and environment simulations.
Significance
This work addresses the critical challenge of achieving high-quality, low-latency multichannel speech enhancement with resource constraints. Its innovative combination of neural networks and traditional filtering principles offers a practical solution for real-time applications like smart assistants, teleconferencing, and hearing aids. The resource-efficient design enables deployment on edge devices, reducing hardware costs and power consumption. The framework’s ability to maintain high speech quality while drastically reducing latency and computational demands marks a significant advancement in the field, pushing the boundary toward real-world, scalable speech enhancement systems.
Technical Contribution
The key technical innovations include: • Development of a low-latency (2 ms) multichannel processing framework based on two lightweight RNNs; • Introduction of a trainable neural Wiener filter that jointly optimizes analysis and synthesis transforms during end-to-end training; • Application of Woodbury formula for efficient matrix inversion, ensuring real-time processing stability; • Demonstration that joint end-to-end training of all components yields superior performance compared to sequential or partial training. These contributions enable resource-efficient, high-performance multichannel speech enhancement, bridging the gap between traditional signal processing and deep learning.
Novelty
This research is pioneering in integrating a trainable neural Wiener filter within a low-latency, end-to-end multichannel speech enhancement framework. Unlike prior works that rely on fixed or separately optimized filters, this approach trains the analysis and synthesis transforms jointly with neural networks, ensuring minimal nonlinear distortion and optimal resource utilization. The combination of two lightweight RNNs with a trainable NWF, optimized via the Woodbury formula, represents a novel architecture that achieves state-of-the-art performance at only 2 milliseconds latency. This is the first comprehensive effort to realize such a resource-efficient, low-delay system with joint training and adaptive filtering in the time domain.
Limitations
- While the model performs well in simulated environments, its robustness in real-world scenarios with highly dynamic noise or multiple moving sources remains to be validated. The training data may not fully capture all environmental variability.
- The current design targets a 2 ms latency; further reduction to sub-millisecond levels might compromise performance, necessitating new architectural innovations.
- Despite parameter and computational savings, deployment on extremely resource-constrained devices (e.g., wearables) may still face challenges, especially under complex acoustic conditions. Future work should focus on adaptive mechanisms and hardware-aware optimization.
Future Work
Future research will explore enhancing robustness against diverse real-world acoustic conditions, including dynamic noise and reverberation. Incorporating adaptive mechanisms for environment-aware parameter tuning could further improve performance. Extending the framework to multi-speaker separation and integrating visual cues for multimodal enhancement are promising directions. Additionally, optimizing the architecture for ultra-low latency (<1 ms) and deploying on embedded hardware will be key steps toward commercial applications. The goal is to develop universally robust, real-time speech enhancement systems suitable for a wide range of practical scenarios.
AI Executive Summary
The demand for real-time, high-quality speech enhancement in noisy environments has driven significant research in multichannel audio processing. Traditional methods like MVDR and GEV beamformers, while effective, suffer from limitations such as high computational complexity and numerical instability, especially under low-latency constraints. Deep learning approaches have advanced the field, but many models are computationally intensive and not suitable for deployment on resource-limited devices.
In this context, the paper introduces an innovative low-latency, resource-efficient framework that combines the strengths of traditional filtering and modern neural networks. The core components include two lightweight recurrent neural networks (LLRNNs) operating in the time domain, and a neural Wiener filter (NWF) that is jointly trained with analysis and synthesis transforms. This design achieves an impressive 2-millisecond algorithmic latency, making it suitable for real-time applications such as smart assistants, teleconferencing, and hearing aids.
The neural Wiener filter is a key novelty. Unlike conventional Wiener filters that rely on fixed or separately optimized coefficients, the NWF in this framework is trainable end-to-end. It uses the analysis transform to estimate filter coefficients based on the enhanced speech target from DNN1, then applies these coefficients to the multichannel noisy input to produce a spatially filtered, less distorted signal. The synthesis transform reconstructs the time-domain waveform. The entire system is trained jointly, ensuring optimal synergy among components.
Experimental results on the DNS dataset demonstrate the effectiveness of the approach. The proposed model outperforms baseline models like MC-Conv-TasNet and UXNet in terms of STOI, PESQ, and SI-SDR, with improvements of over 4% in STOI and significant gains in speech quality scores. The model parameters are reduced by nearly half, and computational complexity is halved compared to comparable models, confirming its resource efficiency.
This work addresses a critical gap in low-latency, resource-efficient multichannel speech enhancement, providing a practical solution for real-world deployment. Its innovative combination of neural networks, traditional filtering principles, and efficient matrix computations paves the way for future advancements in intelligent audio processing. Moving forward, efforts will focus on robustness in diverse environments, multi-speaker separation, and hardware optimization, aiming to realize truly ubiquitous, real-time speech enhancement systems.
Deep Analysis
Background
Multichannel speech enhancement has evolved from classical spatial filtering techniques like MVDR and GEV, which rely on matrix inversion and spectral estimation, to deep learning-based methods that leverage neural networks for direct mapping. Early neural approaches focused on frequency-domain processing, such as spectral masking and beamforming, achieving notable improvements but often at the cost of high latency and computational load. Recent trends include end-to-end time-domain models like TasNet and Conv-TasNet, which reduce latency but face challenges in multi-microphone scenarios. Neural Wiener filters (NWF) have emerged as a promising hybrid approach, combining traditional signal processing with trainable parameters. However, most existing solutions either lack low-latency capabilities or are computationally heavy, limiting their deployment in real-time applications. This paper situates itself at the intersection of low-latency processing, resource efficiency, and multichannel robustness, aiming to bridge the gap between high-performance and practical feasibility.
Core Problem
The core challenge in multichannel speech enhancement is achieving high-quality noise suppression with minimal delay and computational cost. Traditional frequency-domain filters like MCWF are limited by numerical instability and fixed coefficients, which do not adapt well to dynamic environments. Deep neural networks have improved performance but often require large models and high latency, making real-time deployment difficult. Existing low-latency models either compromise on noise suppression quality or are too resource-intensive. The key bottleneck is designing a system that can operate with just a few milliseconds of latency, while maintaining robustness and accuracy across diverse acoustic conditions. This problem is critical for applications requiring immediate feedback, such as voice assistants and hearing aids.
Innovation
The paper introduces several key innovations: 1) a low-latency (2 ms) multichannel processing framework based on two lightweight RNNs that operate in the time domain; 2) a trainable neural Wiener filter (NWF) that jointly optimizes analysis and synthesis transforms during end-to-end training, reducing nonlinear distortions; 3) application of the Woodbury formula for efficient matrix inversion, ensuring real-time processing stability; 4) a unified training strategy that outperforms sequential or partial training, leading to better synergy and performance. Unlike prior models that treat filtering and enhancement separately, this approach integrates them into a single, optimized pipeline, enabling resource-efficient, high-quality noise suppression suitable for real-time scenarios.
Methodology
- �� Input multichannel noisy speech signals, convert to frames via STFT; • DNN1, based on lightweight LSTM, processes frames to produce an enhanced single-channel speech target; • Train analysis transform to estimate Wiener filter coefficients from DNN1 output; • Apply the neural Wiener filter to the multichannel frequency domain representation, utilizing the estimated coefficients for spatial filtering; • Use the Woodbury formula to compute matrix inverse efficiently, maintaining real-time performance; • Convert the filtered frequency domain signal back to time domain through synthesis transform; • Concatenate the NWF output with the original multichannel input, feed into DNN2 for further refinement; • End-to-end joint training of all modules ensures optimal parameter synergy and low nonlinear distortion.
Experiments
采用Interspeech 2020 DNS挑战数据集，模拟多环境噪声和回声，生成80,000个训练样本，1,600个验证样本，3,200个测试样本。模型配置包括不同隐藏层大小（H=128到512），训练采用Adam优化器，学习率2×10^-4，训练200轮。对比多种模型结构，包括单阶段基线模型、DNN+NWF、联合训练多阶段模型等。评估指标涵盖STOI、PESQ、SI-SDR，计算复杂度用GFLOPs衡量。通过消融实验验证分析变换的作用，比较不同训练策略的效果，确保模型在低延迟和资源限制下的优越性。
Results
在DNS数据集上，最优模型在STOI指标达到89.1%，比传统频域Wiener滤波器提升4.2%；PESQ从1.63提升到2.70，提升65%；SI-SDR从-7.48dB改善到7.0dB，表现出极佳的噪声抑制和语音保真能力。参数量仅为传统模型的56.63%，计算量减少近一半。联合训练策略显著优于逐步训练，模型在保持2毫秒延迟的同时，性能超越多项主流模型。消融分析显示，优化分析和合成变换对性能贡献巨大，端到端训练实现了模型的协同优化，验证了方法的有效性。
Applications
该技术适用于智能助理、远程会议、智能家居等场景，特别在对延迟和资源有限制的设备上表现出色。低延迟保证了实时交互体验，资源效率降低了硬件成本。未来可结合声源定位、多模态信息，提升系统鲁棒性，推动在边缘设备上的部署。还可以扩展到多说话人分离、语音识别等任务，增强整体语音处理能力。
Limitations & Outlook
模型在极端噪声环境或强回声条件下的鲁棒性仍需验证，可能受训练数据多样性不足影响。低延迟模型在某些复杂场景下性能有限，未来需优化结构以满足更高的实时性要求。此外，虽然参数和计算量已大幅减少，但在超低延迟（低于2毫秒）场景中仍存在性能瓶颈，需继续探索更高效的模型设计。
Abstract
We introduce a time-domain framework for efficient multichannel speech enhancement, emphasizing low latency and computational efficiency. This framework incorporates two compact deep neural networks (DNNs) surrounding a multichannel neural Wiener filter (NWF). The first DNN enhances the speech signal to estimate NWF coefficients, while the second DNN refines the output from the NWF. The NWF, while conceptually similar to the traditional frequency-domain Wiener filter, undergoes a training process optimized for low-latency speech enhancement, involving fine-tuning of both analysis and synthesis transforms. Our research results illustrate that the NWF output, having minimal nonlinear distortions, attains performance levels akin to those of the first DNN, deviating from conventional Wiener filter paradigms. Training all components jointly outperforms sequential training, despite its simplicity. Consequently, this framework achieves superior performance with fewer parameters and reduced computational demands, making it a compelling solution for resource-efficient multichannel speech enhancement.
References (20)
A Time-Domain Real-Valued Generalized Wiener Filter for Multi-Channel Neural Separation Systems
Yi Luo
FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing
Yi Luo, Enea Ceolini, Cong Han et al.
A Simple RNN Model for Lightweight, Low-compute and Low-latency Multichannel Speech Enhancement in the Time Domain
Ashutosh Pandey, Ke Tan, Buye Xu
STFT-Domain Neural Speech Enhancement With Very Low Algorithmic Latency
Zhong-Qiu Wang, G. Wichern, Shinji Watanabe et al.
TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation
Zhongqiu Wang, Samuele Cornell, Shukjae Choi et al.
On diagonal loading for minimum variance beamformers
X. Mestre, M. Lagunas
Blind Acoustic Beamforming Based on Generalized Eigenvalue Decomposition
Ernst Warsitz, Reinhold Häb-Umbach
Long Short-Term Memory
Sepp Hochreiter, J. Schmidhuber
Acoustic Beamforming for Hearing Aid Applications
S. Doclo, S. Gannot, M. Moonen et al.
On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environments
Jisi Zhang, Catalin Zorila, R. Doddipatla et al.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, X. Zhang, Shaoqing Ren et al.
Unified Architecture for Multichannel End-to-End Speech Recognition With Neural Beamforming
Tsubasa Ochiai, Shinji Watanabe, Takaaki Hori et al.
Frame-by-Frame Closed-Form Update for Mask-Based Adaptive MVDR Beamforming
T. Higuchi, K. Kinoshita, N. Ito et al.
End-to-End Dereverberation, Beamforming, and Speech Recognition in a Cocktail Party
Wangyou Zhang, Xuankai Chang, Christoph Boeddeker et al.
On Spatial Features for Supervised Speech Separation and its Application to Beamforming and Robust ASR
Zhong-Qiu Wang, Deliang Wang
Compensating noise and reverberation in far-field Multichannel Speaker Verification
Sandipana Dowerah, Romain Serizel, D. Jouvet et al.
Multi-Microphone Complex Spectral Mapping for Speech Dereverberation
Zhong-Qiu Wang, Deliang Wang
The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results
Chandan K. A. Reddy, Vishak Gopal, Ross Cutler et al.
Cited By (8)
Cloud-Boosted Low-Compute Multi-Channel Speech Enhancement
Controlling the Parameterized Multi-channel Wiener Filter using a tiny neural network
Better Exploiting Spatial Separability in Multichannel Speech Enhancement with an Align-and-Filter Network
Ultra low-compute complex spectral masking for multichannel speech enhancement
Two-stage UNet with channel and temporal-frequency attention for multi-channel speech enhancement
FoVNet: Configurable Field-of-View Speech Enhancement with Low Computation and Distortion for Smart Glasses
JointNet: Joint Learning for Simultaneous DOA Estimation and Speech Enhancement in Noisy and Reverberant Environments
FSformer: Sparsely and effectively learning key features for multi-channel speech enhancement