SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigation
SpatialNav uses spatial scene graphs for zero-shot VLN, reaching 57.7% SR on R2R and 64.0% on R2R-CE.
Key Findings
Methodology
The paper introduces a zero-shot VLN setting with full pre-exploration and constructs a hierarchical Spatial Scene Graph (SSG) from Matterport3D point clouds. SpatialNav uses GPT-5.1 together with a roughly 7 m agent-centric spatial map, an eight-direction compass-like panoramic image, and remote object localization around candidate destinations. These modules support coarse room-level reasoning, orientation-consistent visual grounding, and prediction of future observations.
Key Results
- On R2R val-unseen, SpatialNav obtains 57.7% SR and 47.8% SPL, improving over SpatialGPT by 9.3 and 11.7 absolute points, respectively. On REVERIE, it reaches 49.6% SR and 34.6% SPL.
- In continuous navigation, SpatialNav achieves 64.0% SR, 51.1% SPL, and 65.4% nDTW on R2R-CE, exceeding VLN-Zero by 21.6 SR points and 24.8 SPL points. On RxR-CE, it obtains 32.4% SR and 24.6% SPL.
- The ground-truth annotation variant SpatialNav† reaches 59.3%, 50.4%, 68.0%, and 39.0% SR on R2R, REVERIE, R2R-CE, and RxR-CE, showing that annotation quality remains a major performance bottleneck.
Significance
The work reframes the central weakness of zero-shot VLN as missing global spatial priors rather than insufficient language ability. Pre-exploration converts one-time perception into a reusable environmental knowledge base, reducing exploratory detours and resolving ambiguities among multiple instruction-compatible rooms or objects. The results demonstrate that explicit spatial representations can substantially narrow the gap between zero-shot and supervised navigation, with direct relevance to robots operating repeatedly in bounded indoor environments.
Technical Contribution
The SSG combines containment structure, geometric boundaries, room categories, and object labels in a reusable hierarchy of house, floor, room, and object nodes. Rather than exposing the entire graph to an MLLM, SpatialNav retrieves a compact task-relevant region. Its compass-like 3×3 image aligns egocentric perception with the top-down map, while remote object localization supplies categories and distances near future waypoints. Together, these mechanisms enable long-horizon, future-aware action selection.
Novelty
Unlike NavGPT, OpenNav, and SpatialGPT, which mainly depend on online local perception, SpatialNav presents a systematic pipeline of full pre-exploration, structured spatial knowledge construction, and task-conditioned retrieval for zero-shot VLN. Compared with VLN-Zero, it does not maintain only symbolic constraints: it jointly models layout, room semantics, and object-level geometry, making the global memory both spatially explicit and visually meaningful.
Limitations
- Room segmentation depends on strong geometric boundaries such as walls. Open-plan areas can be partitioned incorrectly, so regions larger than 20 square meters still require manual verification, limiting scalability.
- The method assumes complete pre-exploration, reliable SLAM point clouds, and access to GPT-5.1 and SpatialLM. Dynamic scenes, changing furniture, localization drift, inference cost, and dependence on a waypoint predictor may reduce real-world robustness.
Future Work
Future research should develop scalable automatic spatial annotation and reduce manual verification. Important directions include dynamic object tracking, cross-floor reasoning, uncertainty-aware graphs, online map updates, smaller and cheaper MLLMs, and real-robot evaluation under localization noise and low-level control constraints. Broader benchmarks should also test whether the pre-exploration cost is justified across many tasks and environments.
AI Executive Summary
Vision-and-language navigation asks an embodied agent to follow natural-language instructions through indoor environments. Supervised systems learn priors such as the typical relationship between kitchens, dining rooms, and bedrooms from large trajectory corpora. Zero-shot agents including NavGPT and SpatialGPT avoid task-specific training, but they usually reason from a narrow local view. When several bedrooms or directions are compatible with an instruction, the agent lacks the global evidence needed to disambiguate them.
SpatialNav changes the information available to the agent. The environment is first explored, and its Matterport3D point cloud is organized into a Spatial Scene Graph containing house, floor, room, and object entities. During execution, GPT-5.1 receives an agent-centric spatial map, a compass-like panorama assembled from eight views, and remote object descriptions around candidate destinations. The agent can therefore reason not only about what it sees now, but also about the layout and objects likely to appear after an action.
Across R2R, REVERIE, R2R-CE, and RxR-CE val-unseen evaluations, SpatialNav obtains 57.7%, 49.6%, 64.0%, and 32.4% success rates, respectively. On R2R-CE it improves over VLN-Zero by 21.6 SR points and 24.8 SPL points. The ground-truth annotation variant reaches 68.0% SR on R2R-CE, revealing that automatic spatial annotation remains limiting. The study makes a strong case for explicit global memory, while its static-scene assumption, pre-exploration requirement, and dependence on expensive models remain open challenges.
Deep Analysis
Background
VLN has evolved from discrete navigation graphs in Matterport3D to continuous environments with low-level control. R2R, REVERIE, R2R-CE, and RxR-CE enabled progress through supervised methods such as PREVALENT, HAMT, DUET, and ScaleVLN. Recent MLLM agents including NavGPT, OpenNav, and SpatialGPT improve training-free generalization, but mainly use local visual observations and therefore lack the spatial priors learned by supervised models.
Core Problem
Most zero-shot agents perceive only roughly three meters around the current position. They cannot reliably infer distant room layouts, target objects, or the consequences of alternative paths. If multiple locations satisfy the language instruction, local evidence is insufficient for disambiguation, causing inefficient exploration. The technical challenge is to compress a complete 3D environment into a structured, retrievable, orientation-consistent representation that an MLLM can use without being overwhelmed.
Innovation
First, the paper defines a zero-shot setting with full pre-exploration, allowing environmental knowledge to be reused across episodes. Second, its SSG jointly represents topology, geometry, room semantics, and object categories. Third, SpatialNav introduces three complementary mechanisms: the agent-centric map for coarse spatial reasoning, the compass-like visual representation for coordinate alignment, and remote object localization for future-aware fine-grained grounding. This goes beyond the symbolic global memory of VLN-Zero.
Methodology
- �� Floor segmentation: build a height histogram, apply DBSCAN, and select high-ranking peaks as floors.
- �� Room segmentation: partition enclosed regions using Bobkov et al.’s geometric heuristic; manually verify regions above 20 m².
- �� Semantic annotation: prompt GPT-5 to classify rooms from exploration frames; fine-tune SpatialLM on Matterport3D training scans to predict object boxes and labels.
- �� Map retrieval: localize the agent by its Z and X-Y coordinates, then project nearby rooms within a 7.68 m radius onto a top-down 1024×1024 map.
- �� Visual encoding: arrange eight 256×256 views, sampled every 45° with a 90° field of view, into one 1024×1024 compass image.
- �� Action selection: query objects and distances around navigation-graph viewpoints or continuous-environment waypoints, then combine them with instructions, history, map, and vision in GPT-5.1.
Experiments
Discrete evaluation uses the Matterport3D simulator with R2R and REVERIE val-unseen splits, containing 783 and 1,328 trajectories across 11 scans. Continuous evaluation uses Habitat v0.3.2 with 100 sampled R2R-CE and 200 sampled RxR-CE trajectories. Metrics are TL, NE, OSR, SR, SPL, and nDTW; stopping within three meters counts as success. Baselines include supervised DUET, ScaleVLN, and Efficient-VLN, plus zero-shot NavGPT, SpatialGPT, Smartway, STRIDER, and VLN-Zero. SpatialNav† replaces predicted annotations with Matterport3D ground truth.
Results
On R2R, SpatialNav records 57.7% SR and 47.8% SPL, versus SpatialGPT’s 48.4% and 36.1%. On REVERIE, it reaches 49.6% SR and 34.6% SPL. In R2R-CE it achieves 64.0% SR, 51.1% SPL, and 65.4% nDTW, compared with VLN-Zero’s 42.4% SR and 26.3% SPL. RxR-CE reaches 32.4% SR. SpatialNav† improves R2R-CE to 68.0% SR, confirming that room and object annotation accuracy directly affects navigation.
Applications
The framework is suited to bounded, repeatedly visited indoor spaces: robotic vacuums, warehouse robots, hotel assistants, and domestic service platforms. A deployment can use RGB video and SLAM to build the SSG offline, then retrieve only task-relevant information during execution. It is particularly useful when instructions mention repeated room types, remote objects, or long sequences of spatial actions.
Limitations & Outlook
SpatialNav assumes a static environment, complete pre-exploration, and an available 3D point cloud. People, moved furniture, occlusion, and SLAM drift can corrupt the graph. Open-plan room segmentation remains unreliable and currently requires manual verification for large regions. GPT-5.1 and SpatialLM introduce cost, latency, and reproducibility concerns, while continuous navigation depends on Shi et al.’s waypoint predictor. Future systems need uncertainty modeling, online graph updates, cheaper backbones, and real-robot validation.
Plain Language Accessible to non-experts
Imagine entering a huge school campus for the first time. A normal zero-shot robot is like a student who can see only the corridor directly ahead. If two doors both seem to lead to a science room, the student must guess, walk around, and possibly backtrack. SpatialNav first lets the robot tour the campus and make a reusable directory: which building is on which floor, which rooms connect, and which objects are inside them.
When the task begins, the robot does not memorize every detail. It checks a small map centered on its current position and arranges views from front, right, back, and left like a compass. It can also ask what objects are near a possible next location—for example, whether a hallway leads toward lockers or laboratory benches. This is like knowing what landmark will appear after choosing a route.
The robot can therefore ask both “What do I see now?” and “What will I probably see if I go there?” On R2R it succeeds 57.7% of the time, and on continuous R2R-CE it succeeds 64.0% of the time. The trade-off is that the campus must be explored beforehand, and mistakes in the map or object labels can still mislead the robot.
ELI14 Explained like you're 14
Picture a giant adventure game where the mission says, “Turn right, enter the bedroom, then go to the bathroom and wait by the sink.” If you can see only the few meters in front of your character, but the house has three bedrooms, how do you know which one is correct? That is the problem many zero-shot robots face!
SpatialNav gives the robot a superpower: a scouting round before the mission starts. It explores the house and makes a structured map showing floors, rooms, and objects. During the mission, it uses a small map around itself plus a compass picture made from views in eight directions. So “right” stays “right” even when the robot turns—no confusing camera tricks!
It can also peek at clues near possible future locations. Maybe two routes look reasonable, but only one is near a bed, cabinet, or bathroom sink. That extra clue helps the robot choose the route that makes sense later, not just the route that looks good right now. It is like using a game’s scouting ability before moving your character.
The results are impressive: 57.7% success on R2R and 64.0% on the more realistic R2R-CE benchmark. But there is a catch. The robot must explore the whole house first, and its map can contain mistakes. A future version should update the map while people and furniture move around—then it could become a genuinely helpful home teammate!
Glossary
Vision-and-Language Navigation
A task in which an embodied agent follows natural-language instructions through a visual environment. It requires language grounding, scene understanding, and sequential action planning.
The paper evaluates this task on R2R, REVERIE, R2R-CE, and RxR-CE.
Spatial Scene Graph
A structured graph that represents entities and relations in an environment. In this paper, house, floor, room, and object nodes are connected mainly through containment relations.
SpatialNav builds the graph from Matterport3D point clouds for reusable global reasoning.
Agent-centric Spatial Map
A top-down local map centered on the agent and aligned with its heading. It presents nearby rooms, position, and trajectory while filtering irrelevant global information.
The implementation uses a 1024×1024 map with 0.015 m grid size and a 7.68 m radius.
Remote Object Localization
Retrieving object categories and distances near a candidate future location rather than relying only on objects currently visible. It provides evidence about likely future observations.
SpatialNav applies it to discrete viewpoints and continuous waypoints.
Success weighted by Path Length
SPL combines task success with path efficiency: successful and shorter trajectories receive higher scores. It penalizes unnecessary exploration.
SpatialNav obtains 47.8% SPL on R2R and 51.1% on R2R-CE.
Open Questions Unanswered questions from this research
- 1 Reliable automatic room and object annotation from incomplete or noisy SLAM point clouds remains unresolved; open-plan regions still require manual verification.
- 2 It is unclear how static SSGs should handle people, moved furniture, changing lighting, and other dynamics. Online updates and uncertainty-aware reasoning are needed.
- 3 The paper does not fully quantify MLLM latency, monetary cost, or real-robot control robustness, leaving deployment efficiency and error recovery open.
Applications
Immediate Applications
Domestic robotic vacuums
A vacuum can scan a home with RGB video and SLAM during initial setup, then build room and object knowledge for instructions such as “go to the kitchen” or “wait beside the sofa.” Stable boundaries and sufficiently accurate point clouds are required; the expected benefit is less wandering and better disambiguation.
Hotel and warehouse service robots
Robots in fixed floors or warehouses can construct an SSG before operation and use room, cabinet, shelf, or appliance semantics to choose routes. Remote object descriptions are valuable when several rooms or storage locations look similar, reducing failed searches and unnecessary patrols.
Long-term Vision
Continuously updated household spatial memory
Future robots could maintain and revise an SSG while furniture and people change, supporting cleaning, delivery, retrieval, and care tasks through natural language. Key obstacles include privacy, uncertainty, compute cost, dynamic perception, and safe action under imperfect maps.
Abstract
Although learning-based vision-and-language navigation (VLN) agents can learn spatial knowledge implicitly from large-scale training data, zero-shot VLN agents lack this process, relying primarily on local observations for navigation, which leads to inefficient exploration and a significant performance gap. To deal with the problem, we consider a zero-shot VLN setting that agents are allowed to fully explore the environment before task execution. Then, we construct the Spatial Scene Graph (SSG) to explicitly capture global spatial structure and semantics in the explored environment. Based on the SSG, we introduce SpatialNav, a zero-shot VLN agent that integrates an agent-centric spatial map, a compass-aligned visual representation, and a remote object localization strategy for efficient navigation. Comprehensive experiments in both discrete and continuous environments demonstrate that SpatialNav significantly outperforms existing zero-shot agents and clearly narrows the gap with state-of-the-art learning-based methods. Such results highlight the importance of global spatial representations for generalizable navigation.