NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments
NavMorph employs a self-evolving world model with continuous latent space and scene memory, boosting VLN-CE performance and online adaptation.
Key Findings
Methodology
NavMorph integrates a recurrent state-space model (RSSM) with a scene evolution memory (CEM) to model environmental dynamics continuously. It comprises two main modules: the Environment-aware Navigator (Inference Network), which infers latent states from current and historical observations, and the Foresight Action Planner (Predictive Network), which predicts future latent states and visual embeddings for planning. The CEM dynamically updates scene features during online testing, enhancing memory and adaptability. The training maximizes a variational lower bound with visual reconstruction, action prediction, and KL divergence regularization, enabling continuous environment modeling. Experiments on R2R-CE and RxR-CE datasets demonstrate superior generalization and online adaptation, outperforming baselines in success rate and path efficiency.
Key Results
- On R2R-CE unseen splits, NavMorph improves success rate by over 4% and SPL by 2%, outperforming static models. In RxR-CE, it achieves 58.02% SR and 48.98% SPL, surpassing prior methods especially in novel environments.
- Ablation studies show removing CEM reduces performance by about 3%, confirming its critical role. The model maintains stable performance under environment changes, indicating strong online adaptation.
- Compared to state-of-the-art, NavMorph reduces trajectory length and increases success rate, demonstrating more efficient and robust navigation in continuous environments.
Significance
This work advances VLN-CE by introducing a self-evolving world model capable of continuous environment dynamics and online adaptation. It addresses the key challenge of environmental variability, enabling robots and AI agents to operate reliably in real-world, unpredictable settings. The approach bridges the gap between static pre-trained models and dynamic real-time environments, providing a foundation for practical embodied AI applications such as autonomous robots, smart assistants, and navigation systems. Its ability to model environment changes continuously and adapt online marks a significant step toward more intelligent, flexible navigation agents, with broad implications for robotics and AI research.
Technical Contribution
NavMorph's core innovation lies in combining RSSM with scene evolution memory (CEM), enabling continuous latent space modeling with online self-updates. The model introduces a novel memory mechanism that dynamically integrates scene features without gradient-based updates, improving efficiency. Its multi-objective training combines visual reconstruction, action prediction, and KL regularization, ensuring stable, continuous environment representations. Unlike prior static or discretized models, NavMorph supports real-time online adaptation, significantly enhancing robustness and generalization in VLN-CE tasks. This framework opens new avenues for continuous, adaptive embodied AI systems.
Novelty
This is the first work to integrate a self-evolving scene memory with a continuous latent space model for VLN-CE. Unlike previous methods relying on static pretraining or discrete states, NavMorph dynamically updates its environment understanding during online navigation, enabling real-time adaptation. The combination of RSSM with scene evolution memory provides a novel mechanism for modeling environment dynamics continuously, addressing key limitations of existing approaches and setting a new standard for adaptive embodied navigation.
Limitations
- Despite its online adaptability, NavMorph's performance may degrade in environments with rapid, extreme changes beyond its modeling capacity, due to limited expressive power of the latent space.
- The scene evolution memory, while effective, introduces additional computational overhead, which could impact real-time performance in large-scale or highly dynamic scenarios.
- The current implementation is validated mainly in simulated environments; transferring to real-world robots requires further handling of sensor noise and hardware constraints.
Future Work
Future research will focus on integrating multimodal sensory data, such as audio and tactile inputs, to enrich environmental understanding. Combining reinforcement learning with the current framework could further optimize navigation policies. Additionally, developing more efficient memory update mechanisms and deploying the model on physical robots will be key steps toward real-world applications. Exploring scalable architectures for large, complex environments remains an open challenge.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个复杂的迷宫里玩游戏。传统的方法就像用一张静态地图，告诉你怎么走，但如果迷宫突然变形或者你走错了路，就会迷失方向。NavMorph就像一个聪明的助手，它能不断学习迷宫的变化，记住你走过的路，还能提前猜到下一步会遇到什么。它用一种特殊的“记忆袋”储存场景信息，随时更新，帮你找到最短的路。这样，无论迷宫怎么变，它都能帮你找到出口，变得越来越厉害。这就像有个随时会变的超级导航员，帮你在复杂环境中自由穿梭。
ELI14 Explained like you're 14
你知道在玩一个超级复杂的迷宫游戏吗？有时候迷宫会突然变形，或者你走错路了，普通的导航方法就会迷失方向。NavMorph就像一个聪明的朋友，它可以不断学习迷宫的样子，记住你走过的每一条路，还能提前猜到下一步会发生什么。它用一种特殊的“记忆袋”把重要的场景信息装起来，随时更新，帮你找到最短的路。这样，无论迷宫怎么变，它都能帮你找到出口，变得越来越厉害。就像有个超级助手在你身边，帮你在复杂的迷宫中自由穿行，真酷！
Abstract
Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to execute sequential navigation actions in complex environments guided by natural language instructions. Current approaches often struggle with generalizing to novel environments and adapting to ongoing changes during navigation. Inspired by human cognition, we present NavMorph, a self-evolving world model framework that enhances environmental understanding and decision-making in VLN-CE tasks. NavMorph employs compact latent representations to model environmental dynamics, equipping agents with foresight for adaptive planning and policy refinement. By integrating a novel Contextual Evolution Memory, NavMorph leverages scene-contextual information to support effective navigation while maintaining online adaptability. Extensive experiments demonstrate that our method achieves notable performance improvements on popular VLN-CE benchmarks. Code is available at https://github.com/Feliciaxyao/NavMorph.