Real-Time Target Sound Extraction
Waveformer combines dilated causal convolution and Transformer for real-time target sound extraction, improving SI-SNRi by up to 3.3dB.
Key Findings
Methodology
This paper introduces Waveformer, a hybrid architecture integrating multi-layer dilated causal convolution (DCC) as the encoder to efficiently expand receptive fields with low complexity, and a Transformer decoder for precise mask estimation conditioned on target labels. The model processes small audio chunks at 44.1kHz, with each chunk passing through DCC layers to capture context, then attended by the Transformer decoder, which generates a mask for target extraction. Training optimizes SI-SNRi, with the model achieving a parameter count of 1.1-3.9 million and real-time factor (RTF) of 0.66-0.94, outperforming prior non-streaming models in both accuracy and efficiency.
Key Results
- In single-target extraction, Waveformer surpasses ReSepformer by 2.2-3.3dB in SI-SNRi, with model size reduced to 1.1-3.9M parameters and RTF indicating real-time operation. Multi-target tasks show 1.2-1.4dB improvements across 2-3 classes, demonstrating robustness in multi-source environments.
- Different configurations (E=256/512, D=128/256) consistently outperform baselines, confirming architecture's flexibility. The model maintains high performance at high sampling rates and multiple classes, with stable inference speed.
- The approach significantly reduces computational cost while boosting signal quality, enabling deployment on consumer-grade hardware for applications like hearing aids, teleconferencing, and smart devices.
Significance
This work addresses the critical challenge of achieving high-quality, low-latency target sound extraction suitable for real-time applications. By innovatively combining DCC and Transformer, it overcomes the limitations of existing models that either lack efficiency or scalability. The resulting architecture paves the way for advanced auditory scene analysis in consumer electronics, offering significant improvements in noise filtering and source separation, with broad implications for personalized audio experiences and assistive hearing technologies.
Technical Contribution
The core technical innovation lies in the integration of multi-layer dilated causal convolution with Transformer attention, enabling large receptive fields with low complexity (O(log R)) while maintaining low latency. The model leverages the strengths of convolutional layers for efficient context aggregation and attention mechanisms for precise target modeling. This hybrid design offers a new paradigm for streaming audio processing, balancing performance, computational cost, and latency, and establishing a foundation for future scalable, real-time audio models.
Novelty
This is the first work to combine dilated causal convolution with Transformer in a streaming target sound extraction framework. Unlike prior models that either rely solely on convolutional or Transformer-based architectures, this hybrid approach effectively addresses the trade-off between receptive field size and computational efficiency, enabling real-time operation with high accuracy in multi-source scenarios.
Limitations
- The model's performance degrades in extremely noisy or highly overlapping multi-source environments, due to the inherent difficulty in accurate mask estimation under such conditions.
- Current implementation is primarily validated on single-microphone data; multi-microphone or spatially-aware extensions are yet to be explored.
- Handling ultra-high sampling rates (e.g., 96kHz) or very long audio streams remains challenging, requiring further architectural optimization for broader applicability.
Future Work
Future research will focus on integrating spatial cues from microphone arrays, expanding dataset diversity, and improving robustness under adverse acoustic conditions. Additionally, efforts will be made to reduce latency further, adapt the architecture for higher sampling rates, and deploy on embedded hardware, broadening real-world usability.
AI Executive Summary
Target sound extraction in noisy environments has long been a challenging problem in audio processing. Existing methods often rely on offline, large-block processing, limiting their applicability in real-time scenarios. This paper introduces Waveformer, a novel hybrid neural network architecture that combines dilated causal convolution (DCC) with Transformer attention mechanisms to enable efficient, low-latency streaming extraction of target sounds.
The core innovation lies in using multi-layer DCC as the encoder, which exponentially expands the receptive field with minimal computational overhead, and a Transformer decoder that attends to recent encoded chunks to generate precise masks conditioned on target labels. This design balances the need for large context understanding with the constraints of real-time processing, achieving a latency of approximately 1.45ms at 44.1kHz. Extensive experiments on synthetic datasets demonstrate that Waveformer outperforms prior models like ReSepformer and Conv-TasNet, with SI-SNRi improvements of up to 3.3dB, while reducing model size by 1.2-4 times and maintaining real-time inference speed.
The results highlight the potential for deploying high-quality target sound extraction in consumer devices such as hearing aids, teleconferencing systems, and smart earbuds. The architecture’s flexibility allows adaptation to multi-target scenarios, with stable performance across various configurations. Looking ahead, the authors plan to incorporate spatial information from microphone arrays, extend to higher sampling rates, and optimize for more complex acoustic environments, paving the way for broader real-world applications of this technology.
Deep Analysis
Background
声源分离技术经历了从传统盲源分离到深度学习驱动的目标提取的演变。早期方法如ICA和NMF在静态环境中取得一定成功，但难以应对复杂多声源场景。近年来，Conv-TasNet、Transformer等模型在语音分离中表现出色，推动端到端学习的发展。尤其是注意力机制模型在捕获长距离依赖方面展现优势，但在实时性和多源处理方面仍存在瓶颈。现有研究多集中在离线处理，缺乏高效的流式架构，限制了其在实际设备中的应用。
Core Problem
核心问题在于如何在保证低延迟的同时，提升多声源环境下的提取质量。传统模型依赖大块音频输入，导致延迟高、计算量大，不适合实时应用。现有流式模型在感受野和效率之间难以兼顾，尤其在高采样率下性能下降明显。此外，模型在多类别、多目标场景中的鲁棒性不足，限制了其实际部署。解决方案需要在感受野、计算效率和多目标处理能力之间找到平衡。
Innovation
本研究提出Waveformer架构，创新点包括：1）采用多层扩张因果卷积（DCC）作为编码器，有效扩大感受野，降低复杂度，支持流式处理；2）引入Transformer解码器，通过注意力机制实现目标相关性建模，提升掩码估计精度；3）结合两者优势，设计低延迟（约1.45ms）且高效的流式模型。该架构突破了传统单一模型在感受野和实时性上的限制，为音频流处理提供新思路。
Methodology
- �� 输入：44.1kHz采样率下的短时音频块（如32样本）
- �� 编码：多层扩张因果卷积（DCC）提取上下文特征，感受野指数级扩展，参数少
- �� 掩码生成：Transformer解码器在有限lookahead（前后各一块）条件下，结合目标标签（one-hot或multi-hot）生成掩码
- �� 目标重建：掩码与编码特征相乘，通过反卷积还原目标声源
- �� 训练：采用SI-SNRi作为损失函数，优化模型参数
- �� 流式处理：每次处理一个块，动态更新上下文缓冲区，确保低延迟
- �� 评估：在多个公开数据集上测试，比较参数量、推理速度和信噪比提升
Experiments
采用FSD Kaggle 2018和TAU Urban Acoustic Scenes 2019合成数据集，生成多类别声源混合音频。模型参数在不同配置（E=256/512，D=128/256）下训练，基线为Conv-TasNet和ReSepformer。指标包括SI-SNRi、模型参数、RTF。训练采用90%信噪比和10%SI-SNR的复合损失，训练100轮，验证最佳模型。模型在单目标和多目标任务中表现优异，参数少、速度快，满足实时需求。
Results
在单目标提取中，Waveformer在SI-SNRi上比ReSepformer提升2.2-3.3dB，模型参数减少至1.1-3.9M，推理RTF达0.66-0.94，优于传统模型。多目标任务中，提升1.2-1.4dB，表现出良好的多声源处理能力。不同配置的模型验证了架构的灵活性和高效性，特别是在高采样率和多类别环境下表现稳定。
Applications
该技术适用于智能耳机、助听器、远程会议系统等场景，能实时过滤环境噪声，提取用户关注的声音。只需少量硬件资源，即可实现高质量的声源提取，为个人化音频体验提供技术支撑。未来还可结合空间信息，拓展多麦克风阵列应用，提升空间定位和分离能力。
Limitations & Outlook
模型在极端噪声或多声源重叠严重时性能下降，且主要在单麦克风环境下验证。对超高采样率（如96kHz）或长时段连续音频的适应性有限。未来需优化模型结构，降低延迟，增强鲁棒性，扩展多麦克风和复杂场景的应用能力。
Plain Language Accessible to non-experts
想象你在一个繁忙的厨房里，很多人在同时做饭，声音交织在一起。你想专注听某个朋友说话，但其他声音像锅碗瓢盆的碰撞声一样吵。传统的方法就像用耳朵去听，慢慢分辨每个声音，但很费时间。现在，科学家设计了一台特别的“耳机助手”，它能在你说话时，快速识别出你想听的声音，把其他声音过滤掉。它就像一个聪明的厨师，知道每个声音的特点，迅速找到目标声音，把它放大，让你听得清清楚楚。这种新技术用数学和计算机算法，让机器像人一样聪明，能在瞬间帮你筛选出想要的声音，应用在手机、助听器等设备上，让我们的生活变得更方便、更智能。
ELI14 Explained like you're 14
想象你在学校的操场上，很多人在说话、玩游戏，声音像一大堆杂乱的噪音。你想听你朋友讲的笑话，但周围的声音太大，听不清楚。科学家们发明了一种特别的“超级耳机”，它可以在你说话的同时，帮你只听你想听的声音，就像有个聪明的朋友在帮你过滤噪音一样。这台“超级耳机”用了一些聪明的数学方法，把不同的声音变成数字，然后用特别的算法找出你要听的那一段，把其他的声音都挡掉。这样，你就能清楚听到朋友的笑话，感觉像在安静的房间里一样。这项技术还在不断改进，将来可以让每个人都拥有这样聪明的耳机，帮助我们在嘈杂的环境中更好地交流和学习！
Abstract
We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder. This hybrid architecture uses dilated causal convolutions for processing large receptive fields in a computationally efficient manner while also leveraging the generalization performance of transformer-based architectures. Our evaluations show as much as 2.2-3.3 dB improvement in SI-SNRi compared to the prior models for this task while having a 1.2-4x smaller model size and a 1.5-2x lower runtime. We provide code, dataset, and audio samples: https://waveformer.cs.washington.edu/.