Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation
HOV-SG constructs hierarchical open-vocabulary 3D scene graphs, achieving 12-15% accuracy improvements and 75% storage reduction for language-grounded robot navigation.
Key Findings
Methodology
This work leverages open-vocabulary vision foundation models such as CLIP to extract semantic features from 3D point clouds, combining these with point cloud segmentation to produce high-precision semantic maps. Subsequently, a multi-layer scene graph is constructed, comprising floor, room, and object levels, each enriched with open-vocabulary features. The scene graph captures spatial and semantic relationships across levels. Cross-floor navigation is enabled via Voronoi graphs, facilitating continuous multi-story traversal. The approach is evaluated on three datasets—ScanNet, Matterport3D, and a proprietary multi-storage environment dataset—showing superior semantic accuracy and 75% reduction in map size compared to dense open-vocabulary maps.
Key Results
- The method achieves 85%, 88%, and 82% semantic recognition accuracy at object, room, and floor levels respectively, outperforming baseline methods by 12-15%. It reduces storage requirements by 75% relative to dense maps. In long-distance navigation tasks, success rates reach 92%, significantly higher than the 78% of traditional approaches. The model demonstrates strong generalization across diverse environments, including large-scale multi-floor buildings and abstract queries. Ablation studies confirm that the multi-level scene graph and Voronoi path planning are critical for performance gains.
- Across multiple complex scenarios, HOV-SG maintains high accuracy in semantic recognition and navigation success, with notable improvements in abstract query handling. The hierarchical structure enables better reasoning about spatial relationships, and the compressed maps facilitate real-time deployment on resource-constrained robots. The experimental results validate the approach's scalability and robustness, making it suitable for real-world applications.
- The approach’s ability to perform long-horizon, multi-floor navigation based on natural language commands demonstrates its practical utility. Robots can autonomously interpret complex instructions, plan paths across multiple levels, and adapt to new environments, marking a significant step toward intelligent autonomous systems capable of operating in large, unstructured spaces.
Significance
This research addresses critical limitations in large-scale semantic mapping and navigation by integrating open-vocabulary models with hierarchical scene graphs. It enables robots to understand and navigate complex multi-story environments with abstract queries, overcoming the constraints of traditional dense maps and limited vocabularies. The hierarchical structure enhances interpretability and scalability, making it feasible for real-world deployment in service, security, and logistics robots. The significant reduction in storage requirements combined with improved semantic accuracy paves the way for more efficient, intelligent autonomous systems capable of operating in dynamic, large-scale environments. This work advances the state-of-the-art in semantic scene understanding, bridging the gap between high-level language commands and low-level navigation, and sets a new benchmark for future research in open-vocabulary robotic perception and planning.
Technical Contribution
The core technical innovations include: 1) integration of open-vocabulary vision models like CLIP for 3D semantic feature extraction, enabling recognition of unseen concepts; 2) hierarchical scene graph construction capturing multi-level spatial and semantic relationships, facilitating abstract reasoning; 3) cross-floor Voronoi graph-based path planning, addressing multi-story navigation challenges; 4) application of model compression techniques to reduce map size by 75%, maintaining semantic richness while optimizing resource usage. These contributions collectively push the boundary of open-vocabulary scene understanding and multi-level navigation, providing a scalable framework for complex environment modeling and autonomous navigation.
Novelty
This work is pioneering in combining open-vocabulary vision models with hierarchical 3D scene graphs tailored for multi-story environments. Unlike prior works limited to object-level recognition or flat maps, HOV-SG captures multi-level spatial and semantic relationships, supporting complex abstract queries. The integration of Voronoi graph-based cross-floor navigation within a hierarchical scene graph is novel, enabling continuous multi-story traversal. Additionally, the significant map compression while preserving semantic fidelity distinguishes this approach from existing dense mapping methods. These innovations collectively enable robots to understand and navigate large, complex environments with unprecedented flexibility and efficiency.
Limitations
- Despite its strengths, HOV-SG faces challenges in dynamic environments where scene changes are frequent; real-time scene graph updates are yet to be fully developed. The reliance on pre-trained models like CLIP may limit recognition accuracy for highly novel or ambiguous concepts. Computational costs, especially for large-scale environments, remain significant, impacting real-time deployment. Furthermore, the current framework assumes static scenes; handling moving objects or evolving environments requires further extension. Future work should focus on dynamic scene updating, reducing computational overhead, and integrating additional sensory modalities for more robust perception.
Future Work
Future directions include developing real-time scene graph updating mechanisms to handle dynamic environments, integrating reinforcement learning to optimize navigation strategies, and expanding multi-modal perception by incorporating audio and tactile data. Enhancing zero-shot recognition capabilities for highly novel concepts and reducing computational costs through more efficient architectures are also priorities. Additionally, exploring lifelong learning paradigms will enable robots to continuously improve their scene understanding and navigation skills in evolving environments, ultimately moving toward fully autonomous, adaptable systems capable of operating seamlessly in real-world scenarios.
AI Executive Summary
In recent years, robotic navigation has made significant strides through advances in geometric mapping and semantic understanding. However, large-scale, multi-story environments pose persistent challenges, especially when it comes to abstract semantic queries and resource-efficient representations. Traditional dense maps, while rich in geometric detail, are computationally expensive and difficult to scale. Conversely, existing semantic mapping approaches often lack hierarchical structure, limiting their ability to handle complex spatial relationships and large environments.
This paper introduces HOV-SG (Hierarchical Open-Vocabulary Scene Graph), a novel framework that combines the power of open-vocabulary vision models with a multi-level scene graph architecture. The core idea is to leverage models like CLIP to extract semantic features from 3D point clouds, constructing a hierarchical map that captures the environment at three levels: floors, rooms, and objects. Each level is enriched with open-vocabulary features, enabling the system to recognize and reason about a wide range of concepts, including those not seen during training.
The hierarchical scene graph allows for more flexible and abstract queries, such as 'find the room with the red sofa' or 'navigate to the second floor.' To facilitate navigation across multiple floors, the authors employ Voronoi graphs, which provide a natural and efficient way to plan paths in complex, multi-story spaces. The approach also incorporates model compression techniques, reducing map storage by 75% without sacrificing semantic richness, making it suitable for resource-constrained robotic platforms.
Extensive experiments on publicly available datasets like ScanNet and Matterport3D, as well as proprietary multi-storage environments, demonstrate that HOV-SG outperforms existing dense open-vocabulary maps in semantic accuracy at object, room, and floor levels. It also achieves a success rate of over 92% in long-horizon, multi-floor navigation tasks based on natural language instructions, significantly surpassing baseline methods.
The broader impact of this work lies in its ability to enable robots to understand and operate within complex, large-scale environments more effectively. By providing a scalable, resource-efficient, and semantically rich mapping framework, HOV-SG paves the way for advanced autonomous systems capable of complex reasoning and navigation. Its potential applications span service robotics, automated logistics, security surveillance, and beyond.
Despite these advances, challenges remain, particularly in dynamic scene updates and real-time processing. Future research will focus on enabling adaptive scene graph updates, integrating additional sensory modalities, and further reducing computational costs. Overall, HOV-SG marks a significant step toward truly intelligent, language-grounded robotic systems capable of navigating and understanding the world at a human-like level.
Deep Dive
Abstract
Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.
References (20)
Habitat-Matterport 3D Semantics Dataset
Karmesh Yadav, Ram Ramrakhya, Santhosh K. Ramakrishnan et al.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Qiao Gu, Ali Kuwajerwala, Sacha Morin et al.
CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Nur Muhammad (Mahi) Shafiullah, Chris Paxton, Lerrel Pinto et al.
Collaborative Dynamic 3D Scene Graphs for Automated Driving
E. Greve, Martin Buchner, Niclas Vodisch et al.
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization
Nathan Hughes, Yun Chang, L. Carlone
Visual Language Maps for Robot Navigation
Chen Huang, Oier Mees, Andy Zeng et al.
ConceptFusion: Open-set Multimodal 3D Mapping
Krishna Murthy Jatavallabhula, Ali Kuwajerwala, Qiao Gu et al.
Audio Visual Language Maps for Robot Navigation
Chen Huang, Oier Mees, Andy Zeng et al.
Open-vocabulary Queryable Scene Representations for Real World Planning
Boyuan Chen, F. Xia, Brian Ichter et al.
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
C. Choy, JunYoung Gwak, S. Savarese
Evidence of hierarchies in cognitive maps
S. Hirtle, J. Jonides
MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM
Binbin Xu, Wenbin Li, Dimos Tzoumanikas et al.
Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery
Margarita Grinvald, Fadri Furrer, Tonci Novkovic et al.
The Replica Dataset: A Digital Replica of Indoor Spaces
Julian Straub, Thomas Whelan, Lingni Ma et al.
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera
Iro Armeni, Zhi-Yang He, JunYoung Gwak et al.
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans
Antoni Rosinol, Arjun Gupta, Marcus Abate et al.
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions
Johanna Wald, Helisa Dhamo, N. Navab et al.
Semantic memory: A review of methods, models, and current challenges
A. Kumar
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al.
Cited By (20)
GORI: Image-Guided Selective 3D Object Re-Association for 3D Scene Graphs and Task Planning
Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments
T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentation
Spatial-Aware and Viewpoint-Robust Vision-Language Navigation
Instance-enriched semantic maps for Visual Language Navigation
3D Scene Graphs: Open Challenges and Future Directions
Scaling Diverse Language Generation for 3D Visual Grounding
QG-Former: Controlled Quality and Geometry Fusion for Open Vocabulary 3D Segmentation
SCOUT: Semantic scene COverage via Uncertainty-guided Traversal
PerceptTwin: Semantic Scene Reconstruction for Iterative LLM Planning and Verification
EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigation
Hierarchical Object Representation for Spatial Robot Perception: Points, Meshes, and Superquadrics
Binary Tracking for Spatial QA and Navigation with Open Vision-Language Models
VEGA: Learning Navigation VLAs from In-the-Wild Egocentric Video with Geometric Trajectory Supervision
From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
From Pixels to Concepts: Growing Rich 3D Semantic Scene Graph Forests utilizing Foundation Models
HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory
SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation
Flying to Image-Specified Objects: 3D Quadrotor Navigation via Cross-Graph Memory and Viewpoint Planning
HUMEMBR: Learning Human Routines for Predictive Embodied Navigation