EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigation
EvoMemNav constructs a self-evolving visual-semantic memory graph with budgeted coarse-to-fine policy, achieving 59.6% SR and 38.9 SPL on GOAT-Bench.
Key Findings
Methodology
EvoMemNav builds a Visual-Semantic Memory Graph (VSMGraph) where raw views serve as primary memory nodes, organized hierarchically with semantic cues and topological relations. It employs a budgeted coarse-to-fine policy: the coarse stage compresses candidate regions via structured bucketing and ranking, while the fine stage invokes a lightweight VLM only on shortlisted candidates for verification and stop decision. The framework integrates reflection-driven write-back (RDCMA), updating graph-attached priors after each subtask, enabling continuous self-improvement without retraining. Experiments on GOAT-Bench and HM3D across object, text, and image goals demonstrate consistent performance gains, especially in multi-instance disambiguation, fewer premature stops, and robust zero-shot generalization.
Key Results
- On GOAT-Bench, EvoMemNav achieves 59.6% success rate (SR) and 38.9 SPL, outperforming previous training-free methods by a significant margin, validating its efficiency and robustness.
- On HM3D object-goal navigation, it reaches 59.2% SR and 33.6 SPL, showing strong cross-scene transferability and generalization.
- Ablation studies reveal that each component—VSMGraph, coarse-to-fine policy, and RDCMA—contributes substantially, with performance improvements over 20% in combined metrics.
Significance
This work advances embodied navigation by introducing a structured, self-evolving memory system that overcomes the limitations of sparse scene graphs and costly 3D reconstructions. Its ability to maintain fine-grained visual evidence and adapt dynamically to environmental feedback addresses key challenges in long-horizon, zero-shot tasks. The integration of budgeted decision policies and reflection mechanisms paves the way for more autonomous, scalable, and generalizable embodied agents, impacting both academic research and real-world robotics applications.
Technical Contribution
The core innovation lies in the design of VSMGraph, which preserves raw view evidence and organizes it hierarchically with semantic cues, avoiding expensive 3D reconstructions. The budgeted coarse-to-fine policy reduces VLM calls, improving efficiency. The reflection-driven write-back (RDCMA) mechanism enables training-free, online memory self-evolution by updating environment priors based on navigation outcomes. These contributions collectively enhance the scalability, interpretability, and adaptability of embodied navigation systems.
Novelty
This is the first work to combine a structured, self-evolving visual-semantic memory graph with a budgeted hierarchical decision policy for zero-shot embodied navigation. Unlike prior scene graphs or 3D reconstructions, it maintains raw visual evidence and continuously updates environment priors without retraining, representing a significant step forward in autonomous navigation research.
Limitations
- The approach relies on pre-trained detection and language models, which may limit performance in environments with poor lighting or occlusions. Its scalability is constrained by the candidate budget, potentially missing relevant regions in very large scenes.
- The reflection mechanism, while effective, does not explicitly model dynamic environmental changes, which could affect long-term robustness. Computational costs, although reduced, still pose challenges for real-time deployment in resource-constrained robots.
- Further research is needed to handle highly dynamic or cluttered environments, and to integrate multi-agent cooperation for more complex tasks.
Future Work
Future directions include integrating dynamic environment modeling, multi-agent collaboration, and end-to-end training of the self-evolving memory system. Exploring adaptive budget strategies and more robust priors could further improve scalability and robustness. Additionally, extending this framework to outdoor or highly dynamic scenarios remains an open challenge.
AI Executive Summary
Long-term autonomous navigation in complex, unseen environments remains a formidable challenge. Existing methods often rely on sparse scene graphs or computationally intensive 3D reconstructions, limiting their scalability and robustness. To address these issues, this work introduces EvoMemNav, a novel framework that constructs a self-evolving visual-semantic memory graph (VSMGraph). This graph preserves raw visual evidence as first-class memory nodes, organized hierarchically with semantic cues and topological relations, enabling detailed environmental understanding. The key innovation is a budgeted coarse-to-fine decision policy: the coarse stage compresses candidate regions via structured bucketing, reducing the search space, while the fine stage selectively invokes a lightweight vision-language model (VLM) for targeted verification and stop decisions. Complementing this, the reflection-driven continual memory adaptation (RDCMA) mechanism updates environment priors after each subtask, facilitating online self-improvement without retraining. Extensive experiments on the GOAT-Bench and HM3D datasets demonstrate that EvoMemNav consistently outperforms prior approaches, achieving 59.6% SR and 38.9 SPL on GOAT-Bench, and 59.2% SR on HM3D object-goal tasks. These results highlight significant gains in multi-instance disambiguation, reduced premature stops, and enhanced zero-shot generalization. The approach’s efficiency stems from reducing VLM calls by over 40%, while maintaining detailed, fine-grained environmental representations. This work marks a substantial step toward scalable, robust, and adaptive embodied agents capable of long-horizon navigation in diverse, real-world settings. Future research will focus on dynamic environment adaptation, multi-agent systems, and end-to-end training of the self-evolving memory system, aiming to further push the boundaries of autonomous embodied intelligence.
Deep Dive
Abstract
Building memory is essential for long-horizon planning in zero-shot embodied navigation. Detector-centric scene graphs often compress observations into sparse nodes, discarding fine-grained visual evidence and accumulating noise, while 3D reconstruction-based methods remain computationally prohibitive. We present EvoMemNav, an efficient, self-evolving, fine-grained memory framework for zero-shot embodied navigation. EvoMemNav constructs a Visual-Semantic Memory Graph (VSMGraph) that keeps raw views as first-class memory and organizes them with lightweight semantic cues and topological relations into a room-view-object hierarchy, preserving fine-grained details for disambiguation and Stop verification. To scale to growing memory, we introduce a budgeted coarse-to-fine policy: a coarse stage compresses the search space into promising regions, and a fine stage invokes a VLM only for targeted verification and decision. Beyond static memories, EvoMemNav performs reflection-driven write-back after each subtask, updating graph-attached priors that encode accumulated environmental knowledge to refine future decisions without retraining. Experiments on GOAT-Bench and HM3D across object, text-description, and image-goal modalities show consistent gains in SR/SPL, with better multi-instance disambiguation, fewer premature stops, and stronger zero-shot generalization.