FloorPlan-VLN: A New Paradigm for Floor Plan Guided Vision-Language Navigation
Introduces FloorPlan-VLN, leveraging structured floor plans as global priors, achieving over 60% success rate improvement with concise instructions.
Key Findings
Methodology
The FP-Nav model integrates rasterized floor plans with dual-view spatio-temporal aligned videos, employing auxiliary tasks—region localization, trajectory reasoning, instruction summarization—to enhance cross-modal alignment. The dataset combines over 10,000 trajectories from Matterport3D with 100+ semantically annotated floor plans. The model uses a pre-trained multimodal large language model (Qwen-2.5-VL) for feature fusion, incorporating spatial relationship encoding. Training includes noise simulation and real-world deployment to ensure robustness.
Key Results
- On the new FloorPlan-VLN benchmark, FP-Nav outperforms adapted SOTA baselines with a success rate increase of over 60%, reaching success rates above 80% in unseen environments. It generalizes well, with success rates over 75%. The model maintains stability under simulated noise and real robot tests, demonstrating robustness against floor plan distortions and actuation drift.
- Compared to traditional VLN models, FP-Nav achieves longer trajectories (average 15m vs. 9m), with higher success rates and more concise instructions. Ablation studies confirm the importance of spatial encoding, video alignment, and auxiliary tasks, with performance drops of at least 30% when any component is removed.
- Real-world robot experiments validate the model’s robustness, with successful navigation despite floor plan distortions and actuation errors, indicating practical viability.
Significance
This work advances embodied AI by integrating global spatial priors—structured floor plans—into VLN, reducing reliance on verbose instructions and enabling more natural human-robot interaction. It addresses core challenges in spatial understanding, offering a scalable, efficient approach for autonomous navigation in complex environments. The methodology bridges the gap between high-level semantic reasoning and low-level perception, paving the way for smarter robots in homes, hospitals, and warehouses. Its robustness and generalization demonstrate significant potential for real-world deployment, marking a step toward spatially intelligent embodied agents.
Technical Contribution
The paper introduces FP-Nav, a multimodal large language model-based framework that fuses rasterized floor plans with egocentric observations via dual-view spatio-temporal alignment. It innovatively employs auxiliary tasks—region localization, trajectory reasoning, instruction summarization—to improve cross-modal understanding. The approach leverages pre-trained models, introduces noise robustness, and demonstrates effective generalization across unseen environments. This integration of structured spatial priors with advanced multimodal fusion sets a new benchmark for spatial reasoning in embodied AI.
Novelty
This is the first work to incorporate structured, vectorized floor plans as global priors in VLN, combined with multimodal large language models for high-level spatial reasoning. Unlike prior methods relying solely on visual or local maps, this approach uses a standardized, semantic-rich environment representation, enabling better generalization and interpretability. The dual-view spatio-temporal alignment strategy and auxiliary tasks further distinguish this work, offering a novel solution to the longstanding challenge of global spatial understanding in navigation.
Limitations
- Model performance depends on the quality and completeness of the floor plans; in real-world scenarios with inaccurate or incomplete maps, effectiveness may decline.
- Current approach primarily supports single-floor navigation; extending to multi-floor environments requires additional modeling of vertical space and staircase transitions.
- High computational costs for training and inference may hinder deployment on resource-constrained robots, necessitating further optimization.
Future Work
Future research will focus on multi-floor and dynamic environment modeling, integrating real-time perception updates, and reducing computational complexity. Exploring end-to-end learning frameworks and unsupervised map refinement could further enhance robustness and scalability, bringing autonomous spatial reasoning closer to human-level intelligence.
AI Executive Summary
The quest for autonomous robots capable of natural, efficient navigation in complex environments remains a central challenge in embodied AI. Traditional vision-language navigation (VLN) approaches rely heavily on verbose, step-by-step instructions, which impose cognitive burdens and limit interaction fluidity. Recognizing that humans often leverage global spatial schematics—like floor plans—to navigate efficiently, this research introduces a novel paradigm: FloorPlan-VLN.
This approach harnesses structured, semantic floor plans as global priors, enabling robots to interpret concise, goal-oriented instructions with minimal guidance. The core innovation, FP-Nav, integrates rasterized floor plans with egocentric observations through a dual-view spatio-temporal alignment strategy, reinforced by auxiliary tasks such as region localization, trajectory reasoning, and instruction summarization. The model employs pre-trained multimodal large language models, notably Qwen-2.5-VL, to fuse visual and spatial information, achieving a high level of understanding.
Extensive experiments on the newly constructed FloorPlan-VLN dataset—comprising over 10,000 trajectories across 70+ buildings—demonstrate remarkable success. The model outperforms existing state-of-the-art baselines by more than 60% in success rate, especially in unseen environments, validating its generalization capability. Real-world robot deployments further confirm robustness against floor plan distortions and actuation errors, highlighting practical viability.
This work significantly advances spatial reasoning in embodied AI, offering a scalable, interpretable framework that bridges high-level semantic understanding with low-level perception. It opens new avenues for intelligent navigation in homes, hospitals, and warehouses, where structured maps are readily available. Future directions include multi-floor support, dynamic environment adaptation, and computational optimization, aiming to realize fully autonomous, spatially intelligent robots capable of seamless human interaction.
Deep Analysis
Background
空间理解和自主导航是机器人智能的核心问题。早期研究如Habitat中的VLN，强调逐步指令和局部感知，但难以实现高效全局路径规划。近年来，结构化地图、3D重建和预训练多模态模型的发展，推动了全局空间推理的可能性。代表性工作包括MapNet、Semantic Scene Graph和Vision-Language模型，解决了局部感知到全局理解的瓶颈。然而，这些方法多依赖密集地图或复杂重建，限制了实际应用的普适性。
Core Problem
现有VLN模型在复杂环境中表现有限，主要因缺乏全局空间先验，导致路径规划和指令理解困难。繁琐的逐步指令增加认知负担，难以实现自然交互。平面图作为全局空间信息的轻量级表达，未被充分利用。如何有效结合平面图与局部观察，提升模型的空间推理能力，成为亟待解决的问题。此外，跨场景泛化和鲁棒性不足，也限制了实际部署。
Innovation
提出利用矢量化平面图作为全局空间先验，结合多模态大模型实现高效对齐。创新点包括：1）将平面图 rasterization 为图像，结合视觉提示编码空间关系；2）设计双视角时空对齐视频策略，增强局部观察与全局布局的对应；3）引入区域语义标注和辅助任务，提升空间关系理解。模型在训练中模拟噪声，验证鲁棒性。该方案突破了传统VLN对逐步指令的依赖，显著提升跨场景泛化能力。
Methodology
- �� 构建矢量化平面图，标注区域类型与ID，转为图像格式；
- �� 采集Matterport3D轨迹，映射区域ID，生成轨迹序列；
- �� 设计简洁指令模板，指明起止区域与停止条件；
- �� 利用MLLM预训练模型（Qwen-2.5-VL）进行平面图 rasterization 和特征融合；
- �� 采用双视角时空对齐视频，将观察与平面图同步编码；
- �� 设计辅助任务（区域定位、轨迹推理、指令总结）强化跨模态理解；
- �� 训练模型实现多模态对齐与动作预测，验证鲁棒性。
Experiments
采用FloorPlan-VLN数据集，融合R2R和RxR轨迹，划分训练、验证（已见和未见场景）。指标包括成功率、轨迹长度和指令复杂度。对比迁移SOTA模型，进行消融实验验证平面图空间编码、视频对齐和辅助任务的重要性。在模拟噪声和真实机器人部署中测试模型鲁棒性，评估泛化能力。超参数调优确保模型在不同场景下表现一致。
Results
模型在新基准上成功率提升超过60%，未见环境中达75%以上，表现出优异的泛化能力。轨迹平均长度由9米提升至15米，指令简洁度提高。消融实验显示空间编码和视频对齐关键作用，噪声环境下仍保持稳定。机器人实际测试中，模型应对平面畸变和执行偏差表现出较强鲁棒性，验证了实用性。
Applications
该技术适用于智能家居、仓储物流和服务机器人，依赖结构化平面图作为环境基础，结合简洁指令实现自主导航。未来可扩展多层楼层支持，结合动态感知，推动机器人在复杂环境中的自主行动，提升人机交互体验。
Limitations & Outlook
模型对平面图的依赖在畸变或手绘环境中表现有限，跨楼层导航仍需多层平面图支持，计算资源需求较高，实际部署存在挑战。未来需优化模型结构和感知能力，提升适应性。
Plain Language Accessible to non-experts
想象你在一个大工厂里工作，工厂里有很多房间和走廊。你需要找到某个特定的房间，但没有详细的指示，只知道从哪个房间开始，目标房间在哪里，以及什么时候停止。工厂里有一张平面图，显示所有房间和走廊的布局。你可以用这张图快速找到路径，而不用一边走一边问人。这个研究就是让机器人也能用类似的方法，利用一张平面图和简短的指令，快速找到路。它不需要逐步告诉机器人每一步，而是让它看懂大致布局，然后自己决定路线。这就像你用地图找到目的地一样，既省事又高效。
ELI14 Explained like you're 14
想象你在学校里玩寻宝游戏，你有一张学校的平面图，上面标记了教室、操场和走廊。你的任务是从起点教室找到藏宝的房间。以前，你可能需要老师告诉你每一步怎么走，但现在你只看地图，知道大致位置，然后自己走。这个研究让机器人也能用类似的方法：它有一张房子的平面图，只需要简短的指令，比如“从厨房到客厅”，它就能用地图和一些智能技巧，自己找到路。这比以前复杂的逐步指示要方便多了，也更像人类用地图导航。这样，机器人就能更聪明、更自主地在房子里找到目标，像我们一样用地图思考。
Glossary
Vision-Language Navigation (VLN) (视觉-语言导航)
一种结合视觉感知与自然语言理解，让机器人自主在环境中导航的方法。
论文中的VLN任务旨在让机器人用简洁指令在未知环境中找到目标。
FloorPlan-VLN (平面图引导的VLN)
利用结构化平面图作为全局空间先验，结合多模态模型实现高效导航的新范式。
本文提出的创新，用于提升空间推理能力。
Multimodal Large Language Models (MLLM) (多模态大语言模型)
预训练在大规模多模态数据上的深度学习模型，融合视觉、文本信息。
模型如Qwen-2.5-VL用于平面图 rasterization 和多模态融合。
矢量化平面图 (Vectorized Floor Plans)
将建筑平面图转化为边界多边形和语义标签的标准数字表示。
支持模型学习和环境理解。
Spatio-Temporal Alignment (时空对齐)
同步编码局部观察与全局布局，确保多模态信息一致性。
通过双视角视频策略实现。
Open Questions Unanswered questions from this research
- 1 如何在动态环境中实时更新平面图信息，保持导航的准确性和鲁棒性仍未解决。
- 2 跨楼层空间理解和多层平面图的集成是未来的重要研究方向。
- 3 模型在极端畸变或手绘图环境中的表现仍需提升，适应性不足。
Applications
Immediate Applications
智能家居助手
利用平面图和简洁指令，帮助家庭机器人自主导航，提升用户体验，适应不同房型布局。
仓储物流机器人
基于结构化平面图实现仓库路径规划，提高效率，减少人工干预。
Long-term Vision
自主城市导航
结合城市平面图和实时感知，推动无人驾驶和城市服务机器人自主行驶。
Abstract
Existing Vision-Language Navigation (VLN) task requires agents to follow verbose instructions, ignoring some potentially useful global spatial priors, limiting their capability to reason about spatial structures. Although human-readable spatial schematics (e.g., floor plans) are ubiquitous in real-world buildings, current agents lack the cognitive ability to comprehend and utilize them. To bridge this gap, we introduce \textbf{FloorPlan-VLN}, a new paradigm that leverages structured semantic floor plans as global spatial priors to enable navigation with only concise instructions. We first construct the FloorPlan-VLN dataset, which comprises over 10k episodes across 72 scenes. It pairs more than 100 semantically annotated floor plans with Matterport3D-based navigation trajectories and concise instructions that omit step-by-step guidance. Then, we propose a simple yet effective method \textbf{FP-Nav} that uses a dual-view, spatio-temporally aligned video sequence, and auxiliary reasoning tasks to align observations, floor plans, and instructions. When evaluated under this new benchmark, our method significantly outperforms adapted state-of-the-art VLN baselines, achieving more than a 60\% relative improvement in navigation success rate. Furthermore, comprehensive noise modeling and real-world deployments demonstrate the feasibility and robustness of FP-Nav to actuation drift and floor plan distortions. These results validate the effectiveness of floor plan guided navigation and highlight FloorPlan-VLN as a promising step toward more spatially intelligent navigation.