OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environments
OpenGraph introduces a hierarchical 3D graph framework using vision-language models for large-scale outdoor scene understanding, achieving 73.4% IoU without fine-tuning.
Key Findings
Methodology
OpenGraph employs instance detection and caption generation via Recognize Anything Model, Grounding DINO, and SBERT to extract object descriptions from images. It projects these descriptions onto LiDAR point clouds to create object-centric 3D maps. The environment is segmented based on lane graph connectivity, forming a multi-layer hierarchical graph including point cloud, lane, instance, segment, and environment layers. Graph neural networks and minimum spanning trees refine object relationships, while large language models enable open-vocabulary reasoning. This integrated approach supports zero-shot semantic understanding and rapid environment querying in large outdoor scenes.
Key Results
- On SemanticKITTI, OpenGraph achieves an average IoU of 73.4% across 19 classes, outperforming RangeNet++, DeepLabV3, and other baselines without model fine-tuning. It demonstrates superior semantic segmentation accuracy and robustness in outdoor environments, with reduced noise levels and better object delineation.
- The hierarchical graph structure enables efficient environment segmentation, topological analysis, and path planning, facilitating interactive map updates and navigation in complex outdoor scenarios. Quantitative ablation studies confirm that caption features and multi-layer design significantly improve performance.
- Experimental results show that the method excels in open-vocabulary object retrieval, environment understanding, and scene reasoning, with notable improvements over existing methods in large-scale outdoor settings.
Significance
This work advances outdoor scene understanding by integrating vision-language models into a hierarchical graph framework, overcoming the limitations of indoor-focused or small-scale methods. It enables robots to recognize and reason about a vast array of objects and environment structures in complex outdoor scenes, supporting applications like autonomous driving, urban mapping, and human-robot interaction. The zero-shot capability reduces dependence on extensive labeled datasets, accelerating deployment in real-world scenarios and fostering more intelligent, adaptable robotic systems.
Technical Contribution
The paper introduces a novel hierarchical 3D graph structure that combines multi-modal object descriptions with spatial and topological information. It innovatively merges vision-language models with large language models for open-vocabulary reasoning, and employs graph neural networks and MST algorithms for relationship refinement. The framework supports scalable environment modeling, fast retrieval, and interactive querying, representing a significant step forward in outdoor semantic mapping and scene understanding.
Novelty
This is the first work to propose a large-scale outdoor open-vocabulary hierarchical 3D graph representation that integrates vision-language models with environment topology. Unlike prior indoor-focused methods, it emphasizes environment segmentation, topological reasoning, and open-vocabulary object recognition in outdoor scenes, addressing challenges like object repetition, occlusion, and scene complexity with a unified, scalable framework.
Limitations
- The current model's robustness in adverse weather or highly dynamic scenes remains limited, especially under heavy occlusion or motion blur, which can affect object detection and description accuracy.
- High computational and storage demands restrict real-time deployment in extremely large environments, requiring further optimization for efficiency.
- Dependence on multimodal fusion and large language models introduces potential failure points when data quality is compromised or computational resources are constrained.
Future Work
Future efforts will focus on improving real-time processing capabilities, robustness in dynamic and adverse conditions, and extending the framework to multi-task scenarios such as semantic SLAM and active perception. Incorporating reinforcement learning for autonomous decision-making and exploring lightweight model architectures are promising directions to enable deployment in resource-constrained platforms.
AI Executive Summary
OpenGraph presents a pioneering hierarchical 3D graph framework designed for large-scale outdoor scene understanding, integrating vision-language models with environment topology. The approach begins with extracting object instances and their textual descriptions from RGB images using Recognize Anything Model, Grounding DINO, and SBERT. These descriptions are projected onto LiDAR point clouds to create object-centric 3D maps, which are then segmented based on lane graph connectivity. The environment is represented through a multi-layer hierarchical graph, including point cloud, lane, instance, segment, and overall environment layers, each encoding different semantic and topological information.
The core technical innovation lies in combining multi-modal object descriptions with spatial relationships, refined through graph neural networks and minimum spanning trees. Large language models enable open-vocabulary reasoning, allowing the system to recognize and query objects beyond predefined classes. Extensive experiments on SemanticKITTI demonstrate that OpenGraph achieves an average IoU of 73.4%, outperforming traditional methods like RangeNet++ and DeepLabV3, all without model fine-tuning. The hierarchical graph structure supports environment segmentation, topological analysis, and path planning, facilitating interactive map updates and navigation.
This research significantly enhances outdoor scene understanding, enabling autonomous vehicles and robots to operate more intelligently in complex environments. Its zero-shot recognition capability reduces reliance on labeled datasets, accelerating deployment in real-world scenarios. The framework's scalability and robustness open new avenues for urban mapping, intelligent transportation, and human-robot interaction.
Looking ahead, the authors aim to optimize computational efficiency, improve robustness in adverse weather, and extend the framework to dynamic, multi-task environments. The integration of reinforcement learning and lightweight models promises to further advance autonomous perception and decision-making in outdoor settings.
Deep Dive
Abstract
Environment representations endowed with sophisticated semantics are pivotal for facilitating seamless interaction between robots and humans, enabling them to effectively carry out various tasks. Open-vocabulary maps, powered by Visual-Language models (VLMs), possess inherent advantages, including zero-shot learning and support for open-set classes. However, existing open-vocabulary maps are primarily designed for small-scale environments, such as desktops or rooms, and are typically geared towards limited-area tasks involving robotic indoor navigation or in-place manipulation. They face challenges in direct generalization to outdoor environments characterized by numerous objects and complex tasks, owing to limitations in both understanding level and map structure. In this work, we propose OpenGraph, the first open-vocabulary hierarchical graph representation designed for large-scale outdoor environments. OpenGraph initially extracts instances and their captions from visual images, enhancing textual reasoning by encoding them. Subsequently, it achieves 3D incremental object-centric mapping with feature embedding by projecting images onto LiDAR point clouds. Finally, the environment is segmented based on lane graph connectivity to construct a hierarchical graph. Validation results from public dataset SemanticKITTI demonstrate that OpenGraph achieves the highest segmentation and query accuracy. The source code of OpenGraph is publicly available at https://github.com/BIT-DYN/OpenGraph.