DM$^3$-Nav: Decentralized Multi-Agent Multimodal Multi-Object Semantic Navigation
DM³-Nav: Fully decentralized multi-agent semantic navigation with multimodal goals, using intent broadcasting and neighbor frontier selection.
Key Findings
Methodology
DM³-Nav employs a fully decentralized architecture where each robot builds local semantic maps via GOAT, detects targets with open-vocabulary models, and communicates asynchronously with neighbors. The core mechanism combines intent broadcasting with distance-weighted frontier selection to coordinate exploration and task allocation without global maps or central control. Robots exchange map fragments, goal status, and navigation intent, merging information through robust map alignment using ORB-RANSAC. This approach reduces redundant exploration, improves scalability, and supports multimodal goals. Evaluation on HM3Dv0.2 and GOAT-Bench datasets shows success rates exceeding 74%, outperforming centralized baselines. Real-world deployment confirms robustness in office environments.
Key Results
- On HM3Dv0.2, DM³-Nav achieved a success rate of 74.6%, surpassing centralized methods, with path efficiency comparable. On GOAT-Bench, success rate reached 89.9%, with shorter paths than single-agent baselines. In real-world tests, two robots located 8 targets within 14 minutes, demonstrating practical viability. Ablation studies confirmed that neighbor communication and intent sharing significantly boost performance, especially in open-vocabulary, multi-object scenarios.
- The decentralized system scales well with up to four robots, increasing success rate by 40% over single robot. Path lengths slightly increase due to coordination overhead but remain efficient. The approach maintains robustness even with intermittent communication, outperforming existing methods that rely on shared global maps or centralized planning. The multi-object multi-robot SPL (MSPL) metric effectively captures performance in complex multi-target tasks.
Significance
This work advances multi-robot autonomous navigation by removing reliance on centralized maps and coordination, enabling scalable, fault-tolerant systems capable of complex multi-object, multimodal tasks. It addresses key limitations of prior approaches, such as single-point failures and poor scalability, paving the way for robust deployment in real-world environments like warehouses, offices, and urban areas. The implicit coordination mechanism offers a new paradigm for multi-agent collaboration, combining local perception, asynchronous communication, and flexible goal handling, thus significantly impacting both academic research and practical robotics applications.
Technical Contribution
The main technical innovations include: 1) a fully decentralized multi-agent semantic navigation framework; 2) an implicit task allocation mechanism based on intent broadcasting and neighbor frontier selection; 3) integration of open-vocabulary, multimodal goal recognition with GOAT-based semantic mapping; 4) a novel multi-object multi-robot SPL metric (MSPL) for performance evaluation. These contributions enable scalable, robust, and flexible multi-robot systems capable of complex semantic tasks without global synchronization, representing a significant step forward in autonomous multi-agent navigation.
Novelty
This is the first system supporting fully decentralized, multimodal, multi-object semantic navigation with open-vocabulary goals. Unlike prior works relying on centralized maps or single-goal focus, DM³-Nav leverages local peer-to-peer communication and implicit coordination, enabling scalable multi-robot collaboration in complex environments. Its innovative frontier selection and intent broadcasting mechanisms set new standards for robustness and flexibility in multi-agent navigation.
Limitations
- The system's performance degrades under severe communication loss or high latency, as neighbor information becomes outdated. Handling conflicting goals or dynamic environments remains challenging, requiring further algorithmic refinement. Scalability to very large environments or heterogeneous robot teams needs additional validation. Computational costs for map alignment and perception could limit real-time performance in resource-constrained settings.
Future Work
Future research will focus on integrating lightweight perception models to reduce computational load, extending the framework to heterogeneous robot teams including UAVs, and improving robustness under communication failures. Additionally, exploring reinforcement learning for adaptive task allocation and path planning, as well as scaling to larger environments, will be key directions. These advancements aim to make multi-robot semantic navigation more practical and widely applicable.
AI Executive Summary
Multi-robot autonomous navigation has traditionally relied on centralized architectures, which face scalability and robustness challenges. These systems often depend on a global map and a central coordinator, making them vulnerable to single points of failure and limiting their deployment in dynamic, large-scale environments. Recognizing these limitations, recent efforts have explored decentralized approaches, but most are restricted to single-object or categorical goals, and still depend on global information sharing.
This paper introduces DM³-Nav, a fully decentralized multi-agent semantic navigation system capable of handling multimodal, multi-object goals without relying on a central controller or shared global map. The core innovation lies in combining implicit task coordination through intent broadcasting with neighbor frontier selection, enabling robots to explore efficiently while avoiding redundant paths. Each robot maintains a local semantic map built with GOAT, supporting open-vocabulary recognition through models like CLIP and SuperGlue, and communicates asynchronously with nearby robots.
Experimental results demonstrate that DM³-Nav outperforms existing centralized and decentralized baselines on benchmark datasets HM3Dv0.2 and GOAT-Bench, achieving success rates exceeding 74% and 89%, respectively. The system’s robustness is validated in real-world office environments, where two robots successfully locate 8 targets within 14 minutes, solely relying on onboard sensing and computation. These findings highlight the potential for scalable, fault-tolerant multi-robot systems capable of complex semantic tasks.
The approach’s significance extends beyond academic novelty, offering practical solutions for warehouse automation, security patrols, and search-and-rescue missions. By removing the dependency on global maps and centralized planning, DM³-Nav paves the way for resilient, adaptable robotic teams operating in unpredictable environments. Future work aims to enhance perception efficiency, incorporate heterogeneous robot types, and improve system robustness under communication constraints, further broadening its applicability and impact.
Deep Dive
Abstract
We present DM$^3$-Nav, a fully decentralized multi-agent semantic navigation system supporting multimodal open-vocabulary goal specification and multi-object missions. In our setting, decentralization implies operation without a central coordinator, global map aggregation, or shared global state at runtime. Robots operate autonomously and coordinate through ad-hoc pairwise communication, exchanging local maps, goal status, and navigation intent without synchronization. An implicit task allocation mechanism combining intent broadcasting and distance-weighted frontier selection reduces redundant exploration while preserving decentralized operation. Evaluations on HM3DSem scenes using the HM3Dv0.2 and GOAT-Bench datasets demonstrate that DM$^3$-Nav matches or exceeds centralized and shared-map baselines while eliminating single points of failure inherent in centralized architectures. Finally, we validate our approach in a real-world office environment using two mobile robots, demonstrating successful deployment relying entirely on onboard sensing and computation. A video of our real-world experiments is available online: https://drive.google.com/file/d/1QiUSCn5rIvtuTUqtuXLPgmt6S8x9-MCZ/view?usp=drive_link