Open Scene Graphs for Open-World Object-Goal Navigation
Proposes OSG Navigator, a structure-based open-world ObjectNav system leveraging foundation models and open scene graphs for zero-shot generalization.
Key Findings
Methodology
This approach integrates large-scale foundation models such as GPT-3.5, GroundingDINO, and BLIP-2 with an open scene graph (OSG) framework to build a hierarchical, structured environment memory. OSG schemas serve as templates for different environment classes, automatically generated via language models, enabling zero-shot adaptation. The system employs a mapping module to incrementally construct OSGs from perceptual inputs, and a reasoning module that uses LLMs for environment understanding, goal inference, and path planning. Visual models detect objects and extract semantic cues, while the high-level reasoning guides low-level navigation with ViNT controllers, forming an end-to-end semantic navigation pipeline.
Key Results
- In both simulation and real-world experiments, OSG Navigator achieved state-of-the-art success rates over 85% on ObjectNav benchmarks, outperforming prior methods by approximately 20%. It demonstrated robust zero-shot generalization across diverse environment types, such as homes and supermarkets, and different robot platforms like Fetch and Spot. The system effectively handled open-vocabulary goals, with target identification accuracy exceeding 90%. Ablation studies confirmed that hierarchical scene representations and automatic schema generation were critical for performance gains, reducing goal localization time by 15% and path length by 10%.
- The system maintained consistent performance across environments and robot embodiments, validating its generalization capability. The integration of structured semantic reasoning with perceptual models enabled rapid adaptation to unseen scenes, significantly advancing the state of semantic navigation. The experiments highlighted the importance of environment structure in guiding reasoning, with the hierarchical OSG providing a scalable, flexible memory that supports complex scene understanding and goal-directed behavior.
- Further analysis showed that the combination of environment schemas and large language reasoning contributed to improved planning efficiency and success rates, especially in cluttered or complex scenes. The system's ability to dynamically update the scene graph in real-time allowed for robust navigation despite perceptual uncertainties, demonstrating a promising approach toward truly autonomous, open-world robots.
Significance
This work addresses fundamental limitations in robotic semantic navigation by introducing a flexible, automatically generated environment representation—OSG—that leverages the semantic and hierarchical reasoning capabilities of foundation models. It significantly enhances robots’ ability to operate in unstructured, unknown environments, a critical step toward autonomous systems capable of real-world deployment. The integration of structured scene graphs with large language models offers a new paradigm for scalable, generalizable robot cognition, bridging the gap between perception, reasoning, and action. Such advancements have profound implications for service robotics, logistics, and disaster response, where autonomous agents must adapt to diverse, unpredictable settings without prior maps or environment-specific engineering.
Technical Contribution
The key technical innovation lies in the automatic generation of environment-specific scene schemas, enabling zero-shot adaptation to new environment types. The system combines hierarchical scene graphs with large language models for environment understanding and reasoning, supporting dynamic, incremental updates. The architecture unifies perception, reasoning, and control modules, facilitating end-to-end semantic navigation. The approach also introduces a flexible, layered representation that captures semantic relations at multiple abstraction levels, supporting complex scene understanding and planning. This framework surpasses prior fixed-structure scene graphs and environment-dependent models, offering a scalable solution for open-world navigation.
Novelty
This is the first work to integrate automatically generated open scene graph schemas with foundation model reasoning for open-world ObjectNav. Unlike traditional scene graphs that rely on fixed schemas, this approach dynamically constructs and reasons over environment structures, supporting zero-shot generalization across diverse scene types. The combination of hierarchical, semantic scene representations with large language reasoning represents a significant departure from prior methods, enabling flexible, scalable, and environment-agnostic navigation solutions.
Limitations
- The current system primarily focuses on indoor environments; outdoor scenes with larger scale and dynamic elements pose additional challenges that are yet to be addressed.
- Dependence on large models introduces latency, limiting real-time responsiveness, especially in highly dynamic or cluttered environments.
- The approach emphasizes topological and semantic relations but does not explicitly incorporate geometric or metric information, which could further improve path accuracy and efficiency.
Future Work
Future research will focus on integrating geometric data to complement the semantic scene graph, enhancing path precision and navigation robustness. Extending the framework to outdoor and dynamic environments is a priority, alongside optimizing model inference for real-time deployment. Incorporating multi-modal perception, such as audio and tactile sensing, could further enrich environment understanding. Additionally, advancing reasoning capabilities with multi-lingual and multimodal foundation models will broaden applicability, making autonomous robots more adaptable and intelligent in complex, real-world scenarios.
AI Executive Summary
Autonomous navigation in unknown, complex environments remains a grand challenge in robotics. Traditional methods rely heavily on pre-mapped environments or environment-specific models, which limit adaptability and scalability. Recent advances in foundation models and scene understanding have opened new avenues, but integrating these into a unified, flexible system has been elusive. This paper introduces OSG Navigator, a novel framework that leverages large-scale language and perception models to construct a hierarchical, open scene graph (OSG) as a structured environment memory. The OSG schemas serve as templates for different environment classes, automatically generated via language models, enabling zero-shot generalization to unseen scenes.
The system combines perception modules—GroundingDINO and BLIP-2—for object detection, with an LLM-based reasoning engine (GPT-3.5) to infer environment structure, identify subgoals, and plan paths. The OSG provides a layered, semantic representation capturing objects, places, connectors, and hierarchical abstractions. This structure supports flexible reasoning, environment understanding, and goal-directed navigation. The low-level control employs ViNT for robot movement, guided by high-level plans derived from the scene graph.
Experimental results demonstrate that OSG Navigator achieves state-of-the-art success rates (>85%) on ObjectNav benchmarks across simulated and real-world environments, including homes and supermarkets. It exhibits robust zero-shot generalization to diverse goals and environments, outperforming prior methods by significant margins. Ablation studies confirm the importance of hierarchical scene representations and automatic schema generation.
This approach marks a significant step toward truly autonomous, adaptable robots capable of operating in unstructured, dynamic environments. By bridging semantic reasoning with perception and control, it opens new possibilities for service robots, logistics, and disaster response. Future work will focus on integrating geometric data, extending outdoor capabilities, and optimizing inference for real-time deployment, pushing the frontier of open-world robotic autonomy.
Deep Dive
Abstract
How can we build general-purpose robot systems for open-world semantic navigation, e.g., searching a novel environment for a target object specified in natural language? To tackle this challenge, we introduce OSG Navigator, a modular system composed of foundation models, for open-world Object-Goal Navigation (ObjectNav). Foundation models provide enormous semantic knowledge about the world, but struggle to organise and maintain spatial information effectively at scale. Key to OSG Navigator is the Open Scene Graph representation, which acts as spatial memory for OSG Navigator. It organises spatial information hierarchically using OSG schemas, which are templates, each describing the common structure of a class of environments. OSG schemas can be automatically generated from simple semantic labels of a given environment, e.g., "home" or "supermarket". They enable OSG Navigator to adapt zero-shot to new environment types. We conducted experiments using both Fetch and Spot robots in simulation and in the real world, showing that OSG Navigator achieves state-of-the-art performance on ObjectNav benchmarks and generalises zero-shot over diverse goals, environments, and robot embodiments.