Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation
Proposed stereo-aware attention and dynamic fusion framework, boosting audio-visual navigation success rate by over 40%.
Key Findings
Methodology
This paper introduces a framework combining a Stereo-Aware Attention Module (SAM) and an Audio-Guided Dynamic Fusion (AGDF) module within a reinforcement learning setup. SAM explicitly models the spatial disparity between left and right audio channels via bidirectional cross-attention, enhancing directional sound perception. AGDF dynamically adjusts the fusion ratio of visual and auditory features based on audio cues, improving robustness to environmental variations. The architecture includes feature extraction, modality fusion, and policy learning components, optimized with PPO. Experiments on Replica and Matterport3D datasets demonstrate superior success rates and path efficiency compared to baselines.
Key Results
- On Replica, success rate reaches 93.2%, SPL 75.5, over 40% improvement over baselines; in unseen environments, success rate is 36.5%, SPL 26.2, indicating strong generalization.
- In absence of visual input, success rate remains at 96.6%, with significantly better path efficiency, validating the effectiveness of spatial and dynamic fusion modules.
- Ablation studies show SAM improves spatial localization, AGDF enhances modality balancing; combined, they significantly boost performance and robustness.
Significance
This work addresses the limitations of static fusion strategies by explicitly modeling stereo spatial cues and adaptively fusing modalities, leading to robust navigation in complex, cluttered, or occluded environments. It advances the state-of-the-art in multimodal perception for embodied AI, with broad implications for autonomous robots in real-world scenarios, including search and rescue, service robots, and assistive devices. The approach enhances both accuracy and environmental adaptability, bridging the gap between simulation and real-world deployment.
Technical Contribution
The core innovations include the SAM module, which explicitly encodes spatial disparity via cross-attention, and the AGDF module, which employs a gating mechanism for real-time modulation of modality contributions. This combination enables the system to better interpret directional cues and adaptively fuse signals, surpassing static or heuristic fusion strategies. The architecture is integrated into a reinforcement learning framework, enabling end-to-end training and real-time decision-making, setting new benchmarks in multimodal navigation.
Novelty
This is the first work to explicitly incorporate stereo audio spatial disparity via a cross-attention mechanism in AVN tasks. Additionally, the dynamic fusion module allows real-time, environment-aware adjustment of modality importance, addressing the challenge of varying environmental conditions. These innovations distinguish this approach from prior static fusion or single-modality methods, representing a significant step forward in embodied multimodal perception.
Limitations
- The model's performance degrades in highly noisy or reverberant environments where stereo disparity cues are obscured or distorted.
- Training relies heavily on simulated datasets; transferring to real-world scenarios with diverse acoustic conditions remains challenging.
- Computational complexity and inference latency may hinder deployment on resource-constrained platforms, requiring further optimization.
Future Work
Future directions include extending the framework to multi-agent scenarios, integrating more sophisticated acoustic scene understanding, and exploring unsupervised or self-supervised learning to reduce reliance on labeled data. Additionally, real-world deployment and robustness testing in diverse environments will be prioritized to facilitate practical applications.
AI Executive Summary
Autonomous navigation in complex environments remains a challenging problem, especially when visual cues are limited or occluded. Traditional methods often rely on static fusion strategies, which fail to adapt to environmental changes and ignore the rich spatial cues embedded in stereo audio signals. This paper presents a novel framework that leverages stereo-aware attention and dynamic modality fusion to address these issues.
The core idea involves explicitly modeling the spatial disparity between the left and right audio channels through a bidirectional cross-attention mechanism, called SAM. This enhances the agent’s ability to perceive sound directionality, even in cluttered or occluded scenes. Complementing this, the AGDF module dynamically adjusts the importance of visual and auditory features based on real-time audio cues, allowing the system to adapt to environmental variations such as noise, reverberation, or partial occlusion.
The entire system is trained using reinforcement learning, specifically the PPO algorithm, to optimize navigation policies. Extensive experiments on the Replica and Matterport3D datasets demonstrate that the proposed method significantly outperforms existing approaches, achieving over 40% improvement in success rate under audio-only conditions. Notably, the model maintains high performance even when visual inputs are unavailable, showcasing its robustness.
These results highlight the importance of explicitly modeling stereo spatial cues and performing deep, adaptive multimodal fusion. The approach opens new avenues for robust, efficient, and generalizable audio-visual navigation, with promising applications in robotics, autonomous vehicles, and assistive technologies. Despite current limitations in noisy environments and computational costs, future work aims to extend the framework to multi-agent systems and real-world deployment, pushing the boundaries of embodied AI.
Deep Dive
Abstract
In audio-visual navigation (AVN) tasks, an embodied agent must autonomously localize a sound source in unknown and complex 3D environments based on audio-visual signals. Existing methods often rely on static modality fusion strategies and neglect the spatial cues embedded in stereo audio, leading to performance degradation in cluttered or occluded scenes. To address these issues, we propose an end-to-end reinforcement learning-based AVN framework with two key innovations: (1) a \textbf{S}tereo-Aware \textbf{A}ttention \textbf{M}odule (\textbf{SAM}), which learns and exploits the spatial disparity between left and right audio channels to enhance directional sound perception; and (2) an \textbf{A}udio-\textbf{G}uided \textbf{D}ynamic \textbf{F}usion Module (\textbf{AGDF}), which dynamically adjusts the fusion ratio between visual and auditory features based on audio cues, thereby improving robustness to environmental changes. Extensive experiments are conducted on two realistic 3D scene datasets, Replica and Matterport3D, demonstrating that our method significantly outperforms existing approaches in terms of navigation success rate and path efficiency. Notably, our model achieves over 40\% improvement under audio-only conditions compared to the best-performing baselines. These results highlight the importance of explicitly modeling spatial cues from stereo channels and performing deep multi-modal fusion for robust and efficient audio-visual navigation.