Towards Array-Invariant Speech Enhancement via Geometry-Aware Dynamic Convolution
Proposes Geo-DConv, integrating microphone geometry into dynamic convolution for array-invariant speech enhancement.
Key Findings
Methodology
This paper introduces a geometry-aware dynamic convolution (Geo-DConv) framework that explicitly incorporates microphone spatial coordinates to adapt convolution kernels dynamically for different array configurations. The core component, the Topology-Aware Coordinate Transformer (TACT), encodes microphone positions using Fourier positional encoding, then employs a Transformer encoder to model global spatial relationships. The resulting transformation matrix guides the generation of convolution kernels, enabling the model to handle arbitrary array geometries. The approach transforms fixed-array models into array-invariant systems, trained on real-world recordings from the RealMAN dataset, effectively bridging the gap between simulated and real environments.
Key Results
- On the RealMAN dataset, the proposed SpatialNet-Geo-DConv and TF-GridNet-Geo-DConv outperform baseline fixed-array models, with SDR improvements of approximately 1.2dB, SI-SDR gains of 1.1dB, and PESQ increases of 0.3 points across various array configurations. Notably, in unseen 6-microphone setups, the models maintain strong generalization, with DNSMOS OVRL scores rising from 1.42 to 2.64, demonstrating robustness to geometric variations.
- Training with randomized microphone numbers and configurations enhances model robustness and array invariance. The Geo-DConv modules reduce parameter count and computational complexity compared to fixed-array counterparts, while achieving comparable or superior performance.
- Experiments confirm the models’ ability to generalize across different array topologies, including unseen configurations, indicating that explicit spatial encoding effectively captures universal spatial patterns, crucial for real-world deployment.
Significance
This work addresses a fundamental limitation in multi-channel speech enhancement: the dependency on fixed microphone array geometries. By explicitly leveraging spatial coordinates, the proposed framework enables models to adapt seamlessly to diverse device configurations, significantly broadening their practical applicability. This approach not only improves robustness and generalization but also reduces the need for device-specific retraining, lowering deployment costs. It paves the way for more flexible, scalable, and intelligent multi-microphone systems in applications like teleconferencing, smart assistants, and hearing aids, ultimately advancing the field toward truly array-invariant speech processing.
Technical Contribution
The key technical innovation lies in integrating spatial geometry into the convolutional kernel generation process via the Geo-DConv module. The combination of Fourier positional encoding and the TACT transformer enables the model to learn a global spatial relationship representation, which guides the dynamic adjustment of convolution kernels. This guarantees array invariance and enhances generalization across different microphone configurations. The framework is compatible with existing fixed-array models, requiring only minimal modifications, thus offering a practical solution for real-world multi-microphone systems.
Novelty
This study is the first to embed explicit spatial geometry information into a deep learning-based dynamic convolution mechanism for speech enhancement, achieving true array-invariance. Unlike prior methods relying on implicit relations or virtual microphone interpolation, this approach models the entire spatial configuration through a transformer-based encoder, providing a comprehensive understanding of the array topology. The permutation-equivariant design ensures stability under arbitrary microphone permutations, marking a significant step forward in multi-channel speech processing.
Limitations
- While effective in typical array configurations, the model’s performance degrades with highly irregular or micro-scale arrays, where spatial encoding may be insufficient. Further refinement of spatial modeling is needed.
- Training relies on large-scale real environment data, which is costly to acquire, and the model’s robustness in extreme acoustic conditions remains to be validated.
- Computational complexity, though reduced compared to fixed-array models, still poses challenges for very large arrays, necessitating further optimization for real-time deployment.
Future Work
Future research will focus on enhancing spatial encoding efficiency, extending the framework to multi-source and multi-speaker scenarios, and integrating source localization for joint enhancement and tracking. Additionally, exploring lightweight architectures and unsupervised training strategies can further improve real-time applicability and robustness in diverse acoustic environments.
AI Executive Summary
Multi-channel speech enhancement has long been constrained by the fixed geometry of microphone arrays, limiting its adaptability across diverse devices and environments. Traditional fixed-array models excel in specific configurations but falter when array geometries change, which is common in real-world applications like smartphones, smart speakers, and hearing aids. Addressing this challenge, the current study introduces a novel framework that explicitly leverages microphone spatial coordinates to achieve array-invariant speech enhancement.
The core innovation is the geometry-aware dynamic convolution (Geo-DConv), which dynamically generates convolution kernels conditioned on the spatial configuration of microphones. This is achieved through a Topology-Aware Coordinate Transformer (TACT), which encodes microphone positions using Fourier positional encoding, then employs a transformer encoder to model global spatial relationships. The resulting transformation matrix guides the kernel adaptation process, enabling the model to handle arbitrary array geometries seamlessly.
Experimental validation on the RealMAN dataset demonstrates that the proposed models, SpatialNet-Geo-DConv and TF-GridNet-Geo-DConv, outperform traditional fixed-array models across multiple configurations. They achieve approximately 1.2dB SDR improvement and maintain strong performance even on unseen 6-microphone setups, with DNSMOS scores rising from 1.42 to 2.64. These results highlight the models’ robustness and generalization capabilities.
This approach significantly advances multi-channel speech enhancement, making it more flexible and deployable in real-world scenarios. By explicitly encoding spatial information, it reduces reliance on device-specific training and enhances performance across diverse array structures. Future work will explore multi-source scenarios, real-time implementation, and further optimization for large-scale arrays, promising a new era of truly array-invariant speech processing.
Deep Dive
Abstract
Multi-channel speech enhancement (SE) systems exhibit superior performance over single-channel methods but are constrained to fixed microphone array configurations. This restricts their real-world deployment across devices with diverse array geometries. While recent array-agnostic SE methods address variable microphone numbers and permutations, they largely fail to exploit explicit array geometry priors when available, missing a crucial cue for optimal spatial filtering. A Geometry-Aware Dynamic Convolution (Geo-DConv) framework is proposed, which explicitly leverages microphone coordinates to transform standard fixed-array SE models into robust array-invariant systems. Experiments are conducted on the recent real-recorded RealMAN multi-channel speech dataset. Results demonstrate that the proposed architecture enables two widely used fixed-array models to adapt to array-invariant settings, with consistent performance improvements across diverse array topologies.