City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs
VoP method leverages web-scale knowledge, boosting ML models' success rate in city navigation from 20-30% to over 60%.
Key Findings
Methodology
This study introduces CityNav, a long-range urban navigation benchmark across four global cities, using Google Street View images. The core innovation is Verbalization of Path (VoP), which prompts models to generate city-scale cognitive maps—highlighting landmarks and directions—thereby enhancing spatial reasoning and decision-making. Various reasoning techniques (e.g., GEPA, chain-of-thought, reflection) are compared, revealing their limitations in dynamic environments. Incorporating VoP results in over 40% success rate improvements, with models autonomously localizing, recognizing landmarks, and planning routes. The evaluation framework directly deploys models on real street graphs, addressing practical challenges like missing links and asymmetric connectivity.
Key Results
- In four cities, state-of-the-art models like GPT-4.1 improved success rates from 20-30% to 60-80%. For example, GPT-4.1's success in New York rose from 15% to 37%, with SPL metrics also significantly improving. The models effectively identify landmarks, interpret city cues, and generate accurate paths. VoP's explicit geographic grounding notably outperforms prior reasoning techniques, especially in long-distance, multi-decision scenarios.
- Static reasoning methods excel in isolated tasks but falter in sequential, real-world navigation. VoP bridges this gap by explicitly extracting and utilizing city-scale knowledge, leading to more reliable route planning. Multilingual capabilities further enhance generalization across diverse urban environments.
- Overall, the approach demonstrates that embedding explicit geographic reasoning within large models enables robust, autonomous city navigation, paving the way for practical applications in robotics and autonomous vehicles.
Significance
This work advances the frontier of AI-driven urban navigation by integrating web-scale knowledge into real-world decision-making. Moving beyond environment-dependent methods, it demonstrates that large multimodal models can internally store and retrieve structured spatial knowledge, enabling autonomous agents to navigate complex cityscapes without explicit maps or annotations. Such capabilities are crucial for deploying intelligent robots and autonomous vehicles in unstructured environments, especially in multilingual, diverse urban settings. The explicit reasoning mechanism (VoP) offers a new paradigm for knowledge-grounded decision-making, bridging static reasoning and dynamic, real-time navigation. This research not only pushes the technical boundaries but also opens new avenues for scalable, knowledge-driven autonomous systems in smart cities.
Technical Contribution
The primary technical innovation is the VoP mechanism, which prompts models to generate explicit city-scale cognitive maps, including landmarks and directional cues. This is complemented by a robust graph correction algorithm that ensures bidirectional connectivity, addressing real-world data inconsistencies. The study constructs a diverse, multilingual dataset (CityNav) with over 50 decision points per route, covering four major cities, to evaluate long-range, knowledge-intensive navigation. The framework integrates multiple reasoning techniques (GEPA, chain-of-thought, reflection) with explicit geographic prompting, significantly outperforming baseline models. The architecture emphasizes internal knowledge extraction, decision traceability, and efficient memory management within a POMDP setting, enabling scalable, real-world deployment.
Novelty
This is the first work explicitly leveraging web-scale knowledge for city navigation via a prompting-based geographic map extraction (VoP). Unlike prior approaches relying on environment annotations or local features, this method prompts models to generate and utilize global spatial maps, significantly improving long-distance decision accuracy. The dataset's diversity and multilingual nature further distinguish this work, addressing real-world complexity. The integration of explicit geographic reasoning into large models represents a novel paradigm shift in autonomous navigation research.
Limitations
- Models still struggle in environments with severe signal loss or highly dense urban canyons, where geographic map extraction is incomplete. The reliance on pre-trained knowledge limits adaptation to entirely new cities or rapidly changing environments.
- Computational costs remain high, especially for multi-round reasoning and large-scale path generation, hindering real-time deployment. The approach also depends on the quality of street view data, which can vary.
- Generalization to unseen cities or dynamic urban scenarios requires further research, including real-time map updating and multimodal sensor fusion.
Future Work
Future directions include integrating real-time perception data for dynamic map updates, extending to robotic platforms with onboard sensors, and improving efficiency for real-time applications. Exploring unsupervised or semi-supervised learning to enhance generalization, and expanding to more diverse cities with different languages and layouts, are also promising. Additionally, combining this approach with autonomous vehicle systems could revolutionize urban mobility in smart cities.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个陌生的城市里迷路了，没有地图，也没有导航软件，只能靠观察周围的标志物，比如一座雕塑、一家咖啡店，或者路牌上的文字。你会用这些线索在脑海中构建一张城市地图，记住每个标志物的位置，然后根据这些信息一步步找到回家的路。这个模型也是一样，它没有提前的完整地图，而是通过观察和记忆街道上的标志物，自己“画”出一张心里的地图，逐步接近目标地点。研究人员让模型学会用这种“脑海地图”导航，结果在复杂的城市环境中表现得非常好，就像你变成了一个会用观察和记忆自主导航的“城市专家”。
ELI14 Explained like you're 14
想象你在一个大城市里玩寻宝游戏，没有GPS，也没有地图，只能靠观察街道和记住一些标志物，比如那家特别的咖啡店或雕塑。你要自己想办法找到终点。这个模型也是一样，它没有提前知道整个城市的地图，而是通过观察街道上的标志物，自己“画”出一张心里的地图，然后一步步走到目的地。研究人员教会它用这种“脑海地图”来导航，就像你用观察和记忆找到路一样。结果，这个方法让模型在复杂的城市环境中表现得更好，能自主找到目标地点，就像你变成了城市里的“导航高手”。
Abstract
Leveraging multimodal large language models (MLLMs) to develop embodied agents offers significant promise for addressing complex real-world tasks. However, current evaluation benchmarks remain predominantly language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. To bridge this critical gap, we introduce the task of Sparsely Grounded Visual Navigation, explicitly designed to evaluate the sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environment. We operationalize this task with CityNav, a comprehensive benchmark encompassing four diverse global cities, specifically constructed to assess raw MLLM-driven agents in city navigation. Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points without additional environmental annotations or specialized architectural modifications. Crucially, agents must autonomously achieve localization through interpreting city-specific cues and recognizing landmarks, perform spatial reasoning, and strategically plan and execute routes to their destinations. Through extensive evaluations, we demonstrate that current state-of-the-art MLLMs, reasoning techniques (e.g., GEPA, chain-of-thought, reflection) and competitive baseline PReP significantly underperform in this challenging setting. To address this, we propose Verbalization of Path(VoP), which explicitly grounds the agent's internal reasoning by probing city-scale cognitive maps (key landmarks and directions toward the destination) from the MLLM, substantially enhancing navigation success. Project Webpage: https://dwipddalal.github.io/AgentNav/