Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement
Proposes SlowFast framework with a state space model for 2ms low-latency speech enhancement, reducing computation by 70%.
Key Findings
Methodology
The proposed SlowFast architecture comprises two branches: a slow branch analyzing the acoustic environment at low frame rate, and a fast branch performing time-domain speech enhancement at high frame rate. The fast branch employs a lightweight state space model (SSM), where its state transition process is dynamically modulated by the slow branch's output. The slow branch, implemented with GRUs and fully connected layers, extracts environmental features and generates modulation parameters ϵS. These parameters regulate the state transition matrix ASj (diagonal for simplicity) and gating gSj in the fast branch, which models the speech signal dynamics. The fast branch's output is mapped back to the time domain for enhancement. Multi-task loss functions, including spectral MSE, SISNR, PESQ, and STOI, guide training. Experiments demonstrate a 70% reduction in MACs while maintaining performance at 2ms latency.
Key Results
- On Voice Bank+Demand dataset, the SSMM-based SlowFast network achieves PESQ-NB of 3.12 and SISNR of 16.62 with only 100M MACs/s under 2ms latency, outperforming baseline single-branch models with 126M MACs/s.
- In the sample-level latency scenario (62.5μs), the model requires only one computation per 1ms, drastically reducing computational load and enabling real-time edge deployment.
- Among different modulation strategies, SSMM shows superior robustness and stability, with minimal performance degradation at lower modulation frequencies, confirming its effectiveness in low-latency settings.
Significance
This work addresses the critical challenge of achieving high-quality speech enhancement under extremely low latency and limited computational resources. By integrating a slow environmental analysis branch with a fast, dynamically modulated enhancement branch, it enables real-time processing on edge devices without sacrificing performance. The approach opens new avenues for deploying low-latency speech enhancement in practical applications such as hearing aids, communication devices, and immersive VR. Its ability to operate at sample-level latency with significantly reduced MACs marks a breakthrough in the field, bridging the gap between theoretical performance and real-world constraints.
Technical Contribution
The key technical innovation lies in the integration of a lightweight SSM within a dual-branch framework, where the slow branch modulates the fast branch's state transitions dynamically. This design reduces redundant computations by reusing environmental features over multiple fast frames, achieving approximately 70% computational savings. The model supports single-sample latency operation, a first in neural network-based speech enhancement, and introduces multiple modulation strategies (EC, FiLM, SSMM) with SSMM demonstrating the best robustness and performance. The approach fundamentally differs from prior methods by leveraging structured state space models for real-time, low-latency enhancement, enabling efficient and scalable deployment.
Novelty
This is the first work to incorporate structured state space models into low-latency speech enhancement within a dual-branch SlowFast architecture. Unlike previous methods that process each frame independently, this approach dynamically modulates the fast branch's state transitions based on environmental context, significantly reducing computation while maintaining high quality. The achievement of sample-level latency (62.5μs) with such low MACs is unprecedented, marking a new milestone in real-time neural speech processing.
Limitations
- The robustness of the model in highly dynamic or noisy environments still needs validation, as environmental changes could affect the slow branch's feature accuracy.
- Dependence on the slow branch's environmental estimation may introduce errors if the slow analysis is compromised.
- Further optimization is required for deployment on ultra-low-power devices, especially under extreme noise conditions.
Future Work
Future research will explore multi-modal data fusion, adaptive modulation mechanisms, and hardware-aware optimization to further improve robustness and efficiency. Extending the framework to other audio tasks like active noise control and multi-microphone processing is also planned, aiming for broader real-world applicability.
AI Executive Summary
Achieving ultra-low latency in speech enhancement remains a significant challenge, especially when balancing computational efficiency with output quality. Traditional deep learning models, while powerful, often require processing entire frames with high computational costs, making real-time deployment on edge devices difficult. This paper introduces a novel framework called SlowFast, inspired by biological systems, that employs a dual-branch architecture. The slow branch continuously analyzes the acoustic environment at a low frame rate, extracting contextual features, while the fast branch performs high-rate time-domain enhancement using a lightweight state space model (SSM). The key innovation is the dynamic modulation of the fast branch's state transitions based on the slow branch's output, enabling the fast branch to adapt to environmental changes efficiently. This design significantly reduces redundant computations, achieving approximately 70% savings in MACs without sacrificing performance.
The experimental validation on the Voice Bank+Demand dataset demonstrates that the proposed approach can meet a 2ms algorithmic latency requirement while maintaining high perceptual quality (PESQ-NB of 3.12) and intelligibility (SISNR of 16.62). Remarkably, the model operates at a sample-level latency of just 62.5 microseconds, suitable for real-time edge deployment, with only 100 million MACs per second. Different modulation strategies, especially the proposed SSMM, show excellent robustness across various settings, confirming the method’s practicality. This work represents a significant step toward deploying efficient, low-latency neural speech enhancement systems in real-world scenarios, such as hearing aids, communication devices, and immersive virtual reality.
Looking ahead, the authors plan to extend their framework to incorporate multi-modal data and hardware-aware optimizations, further pushing the boundaries of low-latency audio processing. The combination of structured state space models with dual-branch architectures opens new avenues for scalable, real-time neural audio applications, promising a future where high-quality speech enhancement is accessible on resource-constrained devices.
Deep Dive
Abstract
Deep learning-based speech enhancement (SE) methods often face significant computational challenges when needing to meet low-latency requirements because of the increased number of frames to be processed. This paper introduces the SlowFast framework which aims to reduce computation costs specifically when low-latency enhancement is needed. The framework consists of a slow branch that analyzes the acoustic environment at a low frame rate, and a fast branch that performs SE in the time domain at the needed higher frame rate to match the required latency. Specifically, the fast branch employs a state space model where its state transition process is dynamically modulated by the slow branch. Experiments on a SE task with a 2 ms algorithmic latency requirement using the Voice Bank + Demand dataset show that our approach reduces computation cost by 70% compared to a baseline single-branch network with equivalent parameters, without compromising enhancement performance. Furthermore, by leveraging the SlowFast framework, we implemented a network that achieves an algorithmic latency of just 62.5 μs (one sample point at 16 kHz sample rate) with a computation cost of 100 M MACs/s, while scoring a PESQ-NB of 3.12 and SISNR of 16.62.