AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

Vision-Language Navigation (VLN) has emerged as a pivotal research area in embodied AI, aiming to enable agents to navigate complex environments by grounding natural language instructions in visual perception. Early VLN approaches predominantly relied on constructing explicit topological maps and performing heuristic planning on these graphs, often requiring precise 3D sensing and SLAM systems. Representative works include discrete waypoint-based navigation in simulators, which simplify path planning but suffer from sim-to-real transfer challenges. To address this, continuous environment VLN (VLN-CE) was introduced, allowing agents to perform low-level action predictions for more realistic navigation.

Recently, large-scale pretrained Vision-Language Models (VLMs) have been leveraged to directly map instructions and egocentric observations to actions, eliminating the need for additional sensors and improving generalization. Notable examples include NaVILA and VLN-R1, which utilize end-to-end VLMs for action prediction. However, these models primarily focus on low-level action outputs without explicit reasoning about the agent’s state or task progress, limiting their ability to recover from errors or perform nuanced planning. This gap motivates the integration of self-aware reasoning mechanisms within VLN frameworks.

The core problem addressed is the lack of explicit, explainable reasoning about the agent’s current state and task progress in existing VLN models. While end-to-end VLM-based methods simplify navigation pipelines, they operate as black boxes, predicting actions without assessing navigation progress or detecting deviations from instructions. This deficiency hampers robustness, especially in complex or ambiguous environments, where error correction and high-level planning are crucial. Conversely, heuristic planning methods rely on additional 3D sensors and explicit maps, which are impractical for large-scale pretraining and deployment.

Therefore, the challenge is to develop a VLN framework that integrates self-aware reasoning about navigation context, progress, and errors, within an end-to-end trainable model that does not require extra sensors, enabling both interpretability and robustness.

AwareVLN introduces three core innovations:

1. Structural Self-aware Reasoning Module: Unlike prior methods that either omit reasoning or generate textual explanations detached from actions, AwareVLN implements a sparsely triggered reasoning mechanism at key navigation nodes. This module produces structured triplet outputs—scene description, progress assessment, and next-step plan—providing explicit self-awareness and guiding subsequent action generation. This design balances computational efficiency and reasoning depth.

2. Automatic Data Engine: To train the reasoning module without manual annotations, the authors design an automatic data engine leveraging Habitat simulator’s semantic room labels and ground-truth waypoints. It identifies key reasoning nodes such as subtask completions, path deviations, and stopping errors. Using Qwen-VL-Max, a general VLM, it generates structured reasoning supervision via multi-turn conversational prompts, enabling scalable, high-quality training data generation.

3. Unified Vision-Language Model Architecture: AwareVLN employs a single VLM that jointly performs reasoning and action prediction, allowing mutual enhancement between these tasks. The model incorporates relative positional encoding to maintain temporal context and uses special tokens to orchestrate reasoning and acting modes, ensuring coherent and adaptive navigation decisions.

• �� Input Encoding: The natural language instruction is tokenized, and egocentric RGB observations are uniformly sampled and encoded via a vision encoder. Relative positional encoding between current and last reasoning steps is added to provide temporal grounding.

• �� Unified Reason-Act Model: A single vision-language model θ takes as input the instruction tokens, previous reasoning text tokens, and current visual features, outputting either reasoning text or action logits. Special tokens [REASON] and [ACT] determine the mode.

• �� Sparse Reasoning Trigger: Reasoning is activated only at key navigation nodes identified by the automatic data engine—subtask completion (detected via room category transitions), path deviation (exceeding spatial error thresholds), and stopping errors (incorrect stopping locations).

• �� Structured Reasoning Output: When reasoning is triggered, the model generates a triplet comprising (1) scene description summarizing current observations, (2) progress assessment indicating completed instruction segments and deviations, and (3) a plan for the next navigation phase.

• �� Automatic Data Engine: Navigation trajectories are collected via expert following and DAgger-based policies in Habitat. Key reasoning nodes are automatically identified using room-level semantics and waypoint deviations. For each node, multimodal context is extracted and fed into Qwen-VL-Max with carefully designed prompts to generate structured reasoning supervision.

• �� Training: The model is pretrained on large-scale navigation and vision-question answering datasets, then fine-tuned with reasoning-augmented trajectories from the data engine. Training uses NVIDIA H20 GPUs; inference runs at ~1 FPS on RTX 4090.

Experiments are conducted on the R2R-CE and RxR-CE continuous environment benchmarks within the Habitat simulator, using the Val-Unseen splits to evaluate generalization. Metrics include Success Rate (SR), Success weighted by Path Length (SPL), Navigation Error (NE), and Oracle Success Rate (OS). Baselines include state-of-the-art methods such as NaVILA, VLN-R1, and others, with varying sensor inputs (monocular RGB, panoramic RGB, depth, odometry).

Ablation studies investigate the impact of removing key reasoning nodes (subtask completion, path deviation, stopping error), disabling special tokens controlling reasoning triggers, and altering the reasoning schedule (dense vs. sparse). Real-world evaluations are performed across corridor, home, and office environments with varying task complexities to assess sim-to-real transfer.

Training details include multi-stage pretraining and fine-tuning with reasoning supervision, leveraging large-scale datasets and automatic data generation.

AwareVLN achieves NE 4.02, SR 65.4%, SPL 55.1% on R2R-CE Val-Unseen using only monocular RGB input, outperforming NaVILA (NE 4.32, SR 62.1%) and other baselines. On RxR-CE Val-Unseen, it attains NE 3.95, SR 67.6%, SPL 56.1%, demonstrating robustness to longer instructions and multilingual data. Ablations show that removing subtask completion nodes reduces SR to 52.3%, omitting path deviation nodes lowers SR to 55.1%, and excluding stopping error nodes decreases SR to 60.0%, confirming the necessity of each reasoning component. Disabling special tokens or performing dense reasoning degrades performance, validating the sparse, structured reasoning design. Real-world tests reveal superior navigation error rates across simple and complex tasks, confirming effective sim-to-real generalization.

AwareVLN is directly applicable to indoor service robots, enabling efficient, sensor-minimal navigation with enhanced error correction and task awareness. It can be integrated into intelligent assistants to improve task execution in complex environments. Augmented reality devices can leverage its vision-language navigation capabilities for accurate indoor localization and guidance. Long-term, the framework can be extended to autonomous drone navigation in 3D spaces and disaster rescue robots operating in unknown, dynamic environments, facilitating robust autonomous exploration and decision-making.

AwareVLN’s reliance on monocular RGB input limits 3D perception accuracy, occasionally causing collisions or imprecise stopping. Although reasoning is sparsely triggered, complex or very long-horizon tasks may still incur computational overhead and latency. The automatic data engine depends on simulator-provided semantic annotations and ground-truth waypoints, posing challenges for deployment in unannotated or highly dynamic real-world environments where data generation strategies must be adapted.