Cloud-Boosted Low-Compute Multi-Channel Speech Enhancement
Proposes a cloud-assisted low-compute multi-channel speech enhancement framework using delayed server outputs, layerwise feature boosting, and collaborative Wiener filtering, achieving +3.77dB SI-SDR.
Key Findings
Methodology
This paper introduces a collaborative framework combining a high-capacity server-side model with a lightweight edge model for real-time speech enhancement. The server employs a Transformer-based SpatialNet that processes multichannel time-frequency representations, outputting enhanced spectrograms and intermediate spatial features. These intermediate features are extracted at multiple depths and transmitted to the edge model, modulated via FiLM layers to provide hierarchical guidance. Additionally, the server's delayed enhanced spectrogram is concatenated with the multichannel input to serve as a prior. The edge model, based on TinyGRU, performs initial masking and spatial filtering, leveraging the server's intermediate features and delayed outputs. A novel collaborative multichannel Wiener filter (MCWF) fuses covariance matrices estimated from both models, weighted adaptively based on environmental dynamics. This fusion enhances spatial filtering robustness under communication delays and adverse acoustic conditions. The entire system is trained and evaluated on DNS-Challenge datasets, demonstrating significant performance gains with minimal computational overhead.
Key Results
- On the DNS-Challenge dataset, the proposed system achieves SI-SDR improvements from -4.96dB (noisy input) to 5.74dB under standard conditions and from -10.01dB to 2.33dB under challenging noise levels, outperforming baseline models by over 3.5dB. PESQ and STOI metrics also show consistent gains, with PESQ reaching 2.33 and STOI exceeding 73% in challenging scenarios.
- Compared to a scaled-up TinyGRU-Large model, our approach yields higher SI-SDR (5.74dB vs. 2.75dB) with only 1.5% parameter increase, indicating that knowledge fusion from the server provides benefits beyond mere model capacity expansion.
- The system maintains robust performance at communication delays up to 128ms, with only moderate degradation, demonstrating practical viability in real-world network conditions.
Significance
This work addresses a critical challenge in deploying high-performance speech enhancement on resource-constrained devices. By intelligently leveraging cloud-based models, it bridges the gap between computationally limited edge devices and powerful server models. The proposed framework enables real-time, high-quality speech enhancement even under adverse noise and latency conditions, paving the way for advanced applications in telecommunication, hearing aids, and smart assistants. The fusion of hierarchical features and statistical covariance matrices introduces a new paradigm for resource-efficient spatial filtering, with potential to influence future research in distributed signal processing and edge-cloud AI systems.
Technical Contribution
The key technical innovations include the introduction of delayed server outputs as auxiliary inputs, hierarchical feature modulation via FiLM layers for multi-scale guidance, and a novel covariance matrix fusion strategy within the multichannel Wiener filtering framework. These components collectively enable effective knowledge transfer from a pre-trained server model to a lightweight edge model, significantly improving performance without substantial computational overhead. The adaptive fusion mechanism dynamically balances current and delayed spatial statistics, enhancing robustness against communication latency and environmental variability. The entire pipeline is designed for causality and real-time operation, representing a substantial advancement over existing hybrid and knowledge-boosting methods.
Novelty
This research is the first to systematically integrate delayed server outputs, multi-layer feature modulation, and covariance matrix fusion into a unified low-latency speech enhancement system. Unlike prior work that either relies solely on server outputs or simple model scaling, this approach exploits hierarchical spatial features and statistical fusion to achieve superior robustness and efficiency. The innovative use of FiLM for hierarchical feature transfer and the adaptive covariance fusion within MCWF constitute significant departures from conventional hybrid beamforming or neural mask-based methods, establishing a new state-of-the-art paradigm for resource-efficient, cloud-assisted speech processing.
Limitations
- The system's performance degrades as communication delay exceeds 128ms, limiting its effectiveness in extremely high-latency networks. Further optimization is needed to maintain robustness under such conditions.
- The approach assumes a fixed pre-trained server model; if the server model is biased or outdated, the overall enhancement quality may decline, necessitating periodic updates or online adaptation.
- In environments with highly dynamic or non-stationary noise sources, the covariance fusion strategy may require additional refinement to sustain optimal performance, especially under rapidly changing acoustic scenes.
Future Work
Future research will focus on adaptive communication strategies to mitigate latency effects, online updating mechanisms for server models to handle non-stationary environments, and extending the framework to multi-user and multi-device scenarios. Additionally, exploring end-to-end training of the entire pipeline and integrating more sophisticated statistical fusion techniques could further enhance robustness. The goal is to develop a fully autonomous, scalable system capable of maintaining high speech quality in diverse real-world conditions with minimal latency and computational cost.
AI Executive Summary
The demand for high-quality, low-latency speech communication has surged with the proliferation of wearable devices such as smart glasses and hearables. However, achieving robust speech enhancement within the strict computational and energy constraints of these devices remains a significant challenge. Traditional lightweight models, while efficient, often fall short in complex acoustic environments, especially under low signal-to-noise ratios. Conversely, high-capacity deep neural networks (DNNs) deployed in the cloud can deliver superior performance but are hindered by latency and bandwidth limitations, making real-time processing difficult.
This paper introduces a novel cloud-assisted low-compute speech enhancement framework that bridges this gap. The core idea is to leverage a powerful, pre-trained server-side model—SpatialNet based on transformers—that processes multichannel audio and outputs enhanced spectrograms along with intermediate spatial features. These intermediate features, extracted at multiple depths, serve as hierarchical guidance for a lightweight edge model based on TinyGRU. To address communication delays, the server’s enhanced spectrogram is delayed and concatenated as an auxiliary input, providing the edge model with a prior estimate of the target speech.
Further, the authors propose a layerwise feature boosting mechanism, where server intermediate features modulate the edge model’s internal states via FiLM layers, enabling multi-scale spatial and spectral guidance. The most innovative aspect is the collaborative multichannel Wiener filter (MCWF), which fuses covariance matrices estimated from both server and edge models, weighted adaptively based on environmental dynamics. This fusion enhances the spatial filtering robustness, especially under communication delays and adverse acoustic conditions.
Extensive experiments on the DNS-Challenge dataset demonstrate the effectiveness of this approach. Under standard and challenging noise conditions, the proposed system achieves SI-SDR improvements of 3.77dB and 3.49dB respectively over the baseline, with only 1.5% increase in model parameters and minimal additional computation. The system maintains stable performance at delays up to 128ms, showing strong practical potential for real-world deployment.
Overall, this work offers a significant step forward in resource-efficient, high-performance speech enhancement. By intelligently combining hierarchical feature transfer, statistical covariance fusion, and cloud-edge collaboration, it paves the way for smarter, more capable wearable devices and communication systems. Future directions include optimizing latency resilience, online adaptation, and multi-device scalability, promising a broad impact across telecommunication, assistive hearing, and smart home applications.
Deep Dive
Abstract
Low-latency, low-compute speech enhancement is essential for wearable devices with real-time communication requirements, but strict computational constraints significantly limit on-device performance. Knowledge Boosting has been proposed as an effective approach to improve edge model performance by leveraging a more capable server-side model, but performance gains for speech enhancement have been limited. We propose a collaborative framework incorporating three techniques: (1) delayed server output as additional input, (2) layerwise feature boosting that transfers intermediate server representations to guide edge inference, and (3) collaborative multichannel Wiener filtering, which fuses weighted covariance matrices estimated from both server and edge models for improved beamforming. Experimental results demonstrate that the proposed collaborative framework significantly outperforms the edge-only baseline with minimal additional computational overhead.
References (20)
Ultra low-compute complex spectral masking for multichannel speech enhancement
Ashutosh Pandey, Juan Azcarreta
A Time-Domain Real-Valued Generalized Wiener Filter for Multi-Channel Neural Separation Systems
Yi Luo
SpatialNet: Extensively Learning Spatial Information for Multichannel Joint Speech Separation, Denoising and Dereverberation
Changsheng Quan, Xiaofei Li
TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation
Zhongqiu Wang, Samuele Cornell, Shukjae Choi et al.
Low Bit Rate Binaural Link for Improved Ultra Low-Latency Low-Complexity Multichannel Speech Enhancement in Hearing Aids
Nils L. Westhausen, B. Meyer
On the Importance of Neural Wiener Filter for Resource Efficient Multichannel Speech Enhancement
Tsun-An Hsieh, Jacob Donley, Daniel D. E. Wong et al.
UX-Net: Filter-and-Process-Based Improved U-Net for real-time time-domain audio Separation
Kashyap Patel, A. Kovalyov, I. Panahi
The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results
Chandan K. A. Reddy, Vishak Gopal, Ross Cutler et al.
Three-stage hybrid neural beamformer for multi-channel speech enhancement.
Kelan Kuang, Feiran Yang, Junfeng Li et al.
Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement
Zhong-Qiu Wang, Hakan Erdogan, Scott Wisdom et al.
Toward Universal Speech Enhancement For Diverse Input Conditions
Wangyou Zhang, Kohei Saijo, Zhong-Qiu Wang et al.
Multi-Microphone Complex Spectral Mapping for Speech Dereverberation
Zhong-Qiu Wang, Deliang Wang
Controlling the Parameterized Multi-channel Wiener Filter using a tiny neural network
Eric Grinstein, Ashutosh Pandey, C. Li et al.
STFT-Domain Neural Speech Enhancement With Very Low Algorithmic Latency
Zhong-Qiu Wang, G. Wichern, Shinji Watanabe et al.
FiLM: Visual Reasoning with a General Conditioning Layer
Ethan Perez, Florian Strub, H. D. Vries et al.
Time-domain Ad-hoc Array Speech Enhancement Using a Triple-path Network
Ashutosh Pandey, Buye Xu, Anurag Kumar et al.
Improving Design of Input Condition Invariant Speech Enhancement
Wangyou Zhang, Jee-weon Jung, Shinji Watanabe et al.
Knowledge boosting during low-latency inference
Vidya Srinivas, Malek Itani, Tuochao Chen et al.
Improved Mask-Based Neural Beamforming for Multichannel Speech Enhancement by Snapshot Matching Masking
Ching Hua Lee, Chouchang Yang, Yilin Shen et al.
Deepfilternet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio
Hendrik Schröter, Alberto N. Escalante, T. Rosenkranz et al.