AgentVLN: Towards Agentic Vision-and-Language Navigation
AgentVLN employs VLM-as-Brain with cross-space mapping and active exploration, achieving superior long-horizon navigation with 73.5% success rate on RxR-CE.
Key Findings
Methodology
This paper introduces AgentVLN, modeling VLN as a Partially Observable Semi-Markov Decision Process (POSMDP). It leverages a Vision-Language Model (VLM) as the central 'brain' for high-level reasoning, decoupling perception and planning via a modular skill library. A cross-space representation mapping projects perception-layer 3D topological waypoints into the image plane, generating pixel-aligned visual prompts. This bridges the gap between 2D visual features and 3D spatial understanding. The system incorporates a context-aware self-correction mechanism to recover from occlusions and trajectory errors, and a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) enabling active querying for geometric depth cues. The model is fine-tuned on the AgentVLN-Instruct dataset with dynamic stage routing conditioned on target visibility, enhancing long-horizon navigation performance.
Key Results
- On RxR-CE and R2R-CE benchmarks, AgentVLN achieves success rates of 73.5% and 69.5%, respectively, outperforming SOTA methods by over 9%. It maintains a lightweight 3B parameter size, enabling real-time inference on edge devices like Jetson. The model significantly reduces error accumulation over long trajectories, especially in occluded or ambiguous environments. Ablation studies confirm the importance of cross-space mapping, self-correction, and active querying, each contributing to performance gains. The system's robustness and efficiency demonstrate its practical deployment potential.
- In real-world tests, AgentVLN maintains high success rates and path efficiency, even under challenging conditions such as occlusion, instruction ambiguity, and environment variability. Its ability to perform real-time inference on resource-constrained hardware validates its suitability for embedded robotic applications. The model's modular design allows easy adaptation to different tasks and environments, supporting diverse embodied AI applications from indoor navigation to autonomous exploration.
- Compared to larger models like JanusVLN (8.2B) or DualVLN (7.1B), AgentVLN achieves comparable or superior performance with fewer parameters, highlighting its efficiency. The cross-space mapping and active query mechanisms are key innovations that enable precise spatial grounding and robust long-horizon planning, setting new benchmarks for lightweight embodied navigation systems.
Significance
This work addresses fundamental challenges in embodied AI: bridging the semantic gap between 2D visual perception and 3D spatial reasoning, and maintaining robustness over long trajectories. By decoupling high-level reasoning from perception and planning, AgentVLN offers a scalable, efficient solution suitable for real-world deployment. Its active querying and self-correction strategies mimic human-like navigation, significantly improving accuracy and robustness in unstructured, unseen environments. This advances the field towards truly autonomous embodied agents capable of complex tasks with limited computational resources, bridging the gap between research and practical robotics applications.
Technical Contribution
The paper's key technical contributions include: 1) a novel cross-space representation mapping that explicitly projects 3D topological waypoints into the 2D image plane, enabling pixel-aligned visual prompts; 2) a context-aware self-correction mechanism that dynamically adjusts trajectories based on occlusion and error feedback; 3) the design of QD-PCoT, allowing the agent to actively query for missing geometric cues via natural language, integrating multi-round reasoning into the navigation pipeline; 4) a lightweight, modular architecture that supports real-time inference on edge devices, making the system practical for deployment. These innovations collectively improve spatial understanding, robustness, and efficiency over existing SOTA models.
Novelty
This research is the first to embed a VLM as the central 'brain' in VLN tasks, explicitly bridging 2D visual features with 3D spatial information through a cross-space mapping. The integration of active, query-driven reasoning (QD-PCoT) introduces a meta-cognitive layer, enabling the agent to seek missing spatial cues actively. Unlike prior works relying solely on depth estimation or implicit feature fusion, this approach enhances interpretability, robustness, and generalization in unstructured environments. The modular, lightweight design further distinguishes it from large-scale, resource-intensive models, marking a significant step toward practical embodied AI.
Limitations
- Despite its strengths, the model may still struggle in environments with extreme occlusion or highly ambiguous instructions, where perception cues are insufficient for accurate localization.
- The cross-space mapping relies on precise camera parameters and depth information; errors here can degrade performance, especially in noisy sensing conditions.
- While lightweight, the system's active querying and self-correction introduce additional latency, which may impact real-time responsiveness in highly dynamic scenarios.
Future Work
Future directions include integrating multi-modal sensors such as LiDAR or stereo cameras to improve spatial perception robustness, developing adaptive active querying strategies via reinforcement learning, and extending the framework to multi-agent collaborative navigation. Additionally, exploring unsupervised or semi-supervised training paradigms could further reduce data requirements and improve generalization to diverse real-world environments.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个复杂的迷宫里玩寻宝游戏。你手里有一张地图，但地图上没有标明每个角落的确切位置，也没有深度信息。你需要找到宝藏，但迷宫里有很多遮挡和不同的路径。以前的机器人就像只会沿着一条直线走，遇到障碍就迷路。而这项新技术就像给机器人装上了聪明的大脑，它可以理解空间的结构，把3D的迷宫变成2D的地图，帮它更好地判断自己在哪里。它还能主动问“前面那堵墙离我有多远？”这样的问题，获取更多空间信息。它会不断调整自己的路线，确保能找到宝藏。通过这些聪明的策略，机器人变得更像一个有经验的探险者，能在复杂环境中自主探索、找到目标。这就像你用地图和问路一样，变得更聪明、更可靠了。
ELI14 Explained like you're 14
想象你在操场上玩捉迷藏，你要找到藏起来的朋友。以前的机器人就像只会沿着一条直线走的机器人，遇到障碍就会迷路。而现在，这个新技术让机器人变得更聪明，它可以用眼睛看环境，把三维空间变成二维的图片，然后根据图片判断目标的位置。它还能主动问“那边的树离我有多远？”这样的问题，帮自己更准确地找到目标。它还会在走路时不断修正路线，确保不会偏离。这样，机器人在复杂的环境中也能像人一样聪明地找到目标，不会轻易迷路。是不是很酷？它就像给机器人装上了大脑和问路的能力，将来可以帮我们导览、送东西，甚至在危险的地方工作！
Abstract
Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.