FlexIO: Flexible Single- and Multi-Channel Speech Separation and Enhancement
FlexIO employs prompt vectors and array-agnostic channel communication to enable flexible multi-microphone and multi-speaker speech separation, achieving SDR up to 9.7dB.
Key Findings
Methodology
FlexIO extends the TUSS framework by integrating array-agnostic channel communication mechanisms (such as TAC, cross-attention, co-attention) within a multi-channel prompt-conditional architecture. It encodes multi-microphone mixtures and prompt vectors, employs TF-LOCOFORMER modules for dual-path time-frequency modeling, and exchanges information across channels to enhance separation robustness. The prompt vectors control the number of output speakers dynamically, enabling flexible multi-source separation. The system is trained with permutation invariant loss to optimize SDR, and evaluated across diverse datasets, demonstrating high adaptability and performance.
Key Results
- FlexIO achieves SDR up to 9.7dB on multi-microphone, multi-speaker scenarios, outperforming baseline models like DNN-IVA and TF-GridNet. It maintains robustness on real CHiME-4 data, with significant improvements in SDR, SIR, PESQ, and STOI metrics across conditions.
- Different channel communication mechanisms (TAC, cross-attention, co-attention) consistently improve performance, especially in 5-mic scenarios, with SDR gains of 0.5-0.8dB over models without communication. The system adapts well to unseen array configurations, including 5-mic setups not seen during training.
- In multi-speaker tasks, FlexIO flexibly adjusts output number, achieving SIR above 20dB in 2-3 speaker conditions, with superior separation quality compared to fixed-output models. It demonstrates excellent generalization across datasets and real recordings.
Significance
This work advances multi-microphone speech separation by offering a unified, flexible model capable of handling arbitrary array configurations and speaker counts. It addresses key limitations of fixed-structure models, enabling deployment in real-world scenarios like meetings, smart devices, and surveillance. The array-agnostic design reduces hardware dependency, facilitating scalable and adaptable solutions. The prompt-controlled output adds a new level of controllability, bridging the gap between fixed and variable source separation, and pushing the frontier of multi-source audio processing.
Technical Contribution
FlexIO introduces a novel architecture combining prompt-based speaker control with array-agnostic channel communication mechanisms, including TAC, cross-attention, and co-attention, within a dual-path TF modeling framework. It extends the TUSS system to multi-channel inputs, supporting arbitrary microphone numbers without retraining. The model achieves high performance in diverse conditions, demonstrating superior generalization and controllability. The integration of multiple communication mechanisms offers flexibility and robustness, representing a significant step forward in universal speech separation.
Novelty
This is the first system to support arbitrary combinations of microphone and speaker numbers in a unified model, leveraging prompt vectors for explicit control and array-agnostic channel communication for flexible multi-microphone processing. Unlike prior work limited to fixed array or speaker configurations, FlexIO's design allows seamless adaptation to various real-world scenarios, marking a major innovation in multi-source speech processing.
Limitations
- Performance may degrade in extreme noise environments or with unseen array configurations beyond training data. The system's reliance on accurate speaker number estimation can introduce errors, impacting separation quality.
- Computational complexity and latency remain challenges for real-time deployment, especially with larger models or multiple communication mechanisms. Further optimization is needed for edge devices.
- Current model assumes prior knowledge of the number of speakers; automatic speaker counting remains an open problem for future research.
Future Work
Future directions include developing adaptive speaker counting methods, reducing model complexity for real-time applications, and extending the framework to joint source recognition and separation tasks. Incorporating unsupervised or semi-supervised learning could further enhance robustness in diverse acoustic environments.
AI Executive Summary
In the rapidly evolving field of speech processing, the challenge of effectively separating multiple speakers in diverse acoustic environments remains a critical hurdle. Traditional models often rely on fixed array configurations and predefined speaker counts, limiting their applicability in real-world scenarios where microphone setups and speaker numbers vary dynamically. Addressing this gap, the present work introduces FlexIO, a versatile speech separation and enhancement system capable of handling arbitrary numbers of microphones and speakers simultaneously.
FlexIO builds upon the task-aware unified source separation (TUSS) framework, innovatively integrating array-agnostic channel communication mechanisms such as transform-average-concatenate (TAC), cross-attention, and co-attention. These mechanisms facilitate information exchange across channels regardless of array configuration, enabling the model to adapt seamlessly to different microphone setups. The core of FlexIO is a dual-path time-frequency (TF) modeling architecture, the TF-LOCOFORMER, which processes multi-channel mixture representations along with prompt vectors—learned embeddings that specify the number of target speakers.
The prompt vectors serve as explicit controls, allowing the system to dynamically adjust the number of output streams during inference. This flexibility is crucial for real-world applications like conference transcription, hearing aids, and smart assistants, where the number of speakers is often unknown or variable. The model employs permutation invariant training (PIT) to optimize separation quality, evaluated across datasets including CHiME-4, WSJ0-mix, and WHAMR, demonstrating SDR improvements up to 9.7dB and robust performance on real recordings.
Experimental results confirm that FlexIO outperforms existing fixed-configuration models, maintaining high separation quality across different array sizes and speaker counts. Its array-agnostic design reduces hardware dependency, making it suitable for deployment in diverse environments. While challenges remain in reducing computational load and enhancing automatic speaker counting, this work marks a significant step toward universal, flexible multi-source speech processing, promising broad impact in both research and industry.
Deep Dive
Abstract
Speech separation and enhancement (SSE) has advanced remarkably and achieved promising results in controlled settings, such as a fixed number of speakers and a fixed array configuration. Towards a universal SSE system, single-channel systems have been extended to deal with a variable number of speakers (i.e., outputs). Meanwhile, multi-channel systems accommodating various array configurations (i.e., inputs) have been developed. However, these attempts have been pursued separately. In this paper, we propose a flexible input and output SSE system, named FlexIO. It performs conditional separation using prompt vectors, one per speaker as a condition, allowing separation of an arbitrary number of speakers. Multi-channel mixtures are processed together with the prompt vectors via an array-agnostic channel communication mechanism. Our experiments demonstrate that FlexIO successfully covers diverse conditions with one to five microphones and one to three speakers. We also confirm the robustness of FlexIO on CHiME-4 real data.