FaSNet: Low-latency Adaptive Beamforming for Multi-microphone Audio Processing
FaSNet, a time-domain neural beamformer, outperforms traditional methods with 14.3% WER reduction on CHiME-3.
Key Findings
Methodology
FaSNet employs a two-stage framework: first, it estimates frame-level adaptive filters for a reference microphone using normalized cross-correlation (NCC) features and a temporal convolutional network (TCN). The second stage uses the output to estimate filters for other microphones, leveraging content-dependent features. The filters are convolved with the signals and summed to produce the enhanced output. The entire process operates in the time domain, avoiding frequency domain delays, and supports low-latency applications. The training can optimize SI-SNR or mel-spectrogram errors, making the system flexible for various tasks.
Key Results
- In reverberant speech enhancement, FaSNet surpasses oracle beamformers, achieving SI-SNR improvements of 3-6dB with only 1.5 million parameters. It performs well in low-latency settings (L=16ms).
- On CHiME-3 ASR, trained with frequency-domain objectives, FaSNet reduces word error rate (WER) by 14.3% relative compared to baseline models.
- In multi-microphone setups, FaSNet matches or exceeds the performance of Conv-TasNet, demonstrating its effectiveness for real-time processing with minimal delay.
Significance
This work advances low-latency, adaptive beamforming by integrating deep learning into the time domain, addressing the latency and robustness issues of traditional frequency-based methods. It enables high-quality speech enhancement suitable for real-world applications like hearing aids, smart assistants, and teleconferencing, where quick response and noise robustness are critical. The approach bridges the gap between theoretical performance and practical deployment, paving the way for more intelligent, responsive audio systems.
Technical Contribution
The paper introduces a novel two-stage time-domain neural network that learns content-dependent adaptive filters via NCC features and TCNs. This design reduces system latency, simplifies model complexity, and enhances robustness. It combines spatial filtering with deep learning, enabling end-to-end training for multiple objectives, and demonstrates superior performance over classical and frequency-domain neural beamformers.
Novelty
This is the first to implement a fully time-domain, neural adaptive beamformer with a two-stage filter estimation process based on NCC features. It overcomes the long-segment dependency of frequency-domain methods, providing a content-aware, low-latency solution that adapts to dynamic acoustic environments, a significant leap from prior static or frequency-based approaches.
Limitations
- The model's robustness in highly dynamic or non-stationary noise environments remains to be validated, as training data may not cover all real-world scenarios.
- While computationally efficient, performance may degrade with very large microphone arrays or complex reverberant spaces.
- Current training focuses on signal quality metrics; integrating end-to-end ASR objectives could further improve recognition accuracy but is yet to be explored.
Future Work
Future research will focus on extending FaSNet to handle highly dynamic environments, incorporating end-to-end ASR training, and optimizing for deployment on resource-constrained devices. Exploring larger microphone arrays and more diverse acoustic conditions will also be prioritized to enhance robustness and generalization.
AI Executive Summary
The demand for real-time, high-quality speech processing in noisy environments has driven significant innovations in microphone array technology. Traditional beamforming methods like MVDR and GEV excel in static conditions but struggle with latency and adaptability. Deep learning approaches have improved performance but often rely on frequency domain processing, which introduces inherent delays unsuitable for low-latency applications. To address this, the authors propose FaSNet, a novel time-domain neural beamformer designed for low-latency, adaptive speech enhancement.
FaSNet operates in two stages: first, it estimates adaptive filters for a reference microphone using normalized cross-correlation features and a temporal convolutional network, capturing spatial and content information efficiently. Then, it uses the output to estimate filters for remaining microphones, enabling content-dependent, spatially adaptive filtering. This approach avoids the latency issues of frequency-based methods, making it suitable for real-time applications.
Experimental results demonstrate that FaSNet significantly outperforms traditional oracle beamformers in reverberant environments, with SI-SNR gains of 3-6dB, while maintaining a small model size of 1.5 million parameters. When trained with frequency-domain objectives on the CHiME-3 dataset, FaSNet achieves a 14.3% relative reduction in word error rate, confirming its effectiveness as a front-end for speech recognition systems.
The system's ability to learn content-dependent, adaptive filters in the time domain marks a major step forward in low-latency audio processing. Its robustness across different noise and reverberation conditions suggests broad applicability in consumer electronics, telecommunication, and assistive devices. Future work aims to extend this framework to more dynamic acoustic environments, integrate end-to-end recognition training, and optimize for deployment on embedded platforms, promising a new era of intelligent, responsive audio systems.
Deep Dive
Abstract
Beamforming has been extensively investigated for multi-channel audio processing tasks. Recently, learning-based beamforming methods, sometimes called \textit{neural beamformers}, have achieved significant improvements in both signal quality (e.g. signal-to-noise ratio (SNR)) and speech recognition (e.g. word error rate (WER)). Such systems are generally non-causal and require a large context for robust estimation of inter-channel features, which is impractical in applications requiring low-latency responses. In this paper, we propose filter-and-sum network (FaSNet), a time-domain, filter-based beamforming approach suitable for low-latency scenarios. FaSNet has a two-stage system design that first learns frame-level time-domain adaptive beamforming filters for a selected reference channel, and then calculate the filters for all remaining channels. The filtered outputs at all channels are summed to generate the final output. Experiments show that despite its small model size, FaSNet is able to outperform several traditional oracle beamformers with respect to scale-invariant signal-to-noise ratio (SI-SNR) in reverberant speech enhancement and separation tasks. Moreover, when trained with a frequency-domain objective function on the CHiME-3 dataset, FaSNet achieves 14.3\% relative word error rate reduction (RWERR) compared with the baseline model. These results show the efficacy of FaSNet particularly in reverberant and noisy signal conditions.