Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
Struct2D enables 3D spatial reasoning using perception-guided structured 2D inputs, achieving high performance without explicit 3D data.
Key Findings
Methodology
This work introduces Struct2D, which transforms RGB-D videos into structured 2D inputs—bird's-eye view images with object marks and metadata—using a perception module. Egocentric keyframes are optionally incorporated. These inputs prompt large language models like GPT-o3 for zero-shot spatial reasoning. Building on this, the authors create Struct2D-Set, a dataset of 200,000 QA pairs across eight spatial categories, generated automatically from 3D indoor scene datasets such as ScanNet and ARKitScenes. Fine-tuning Qwen2.5VL on this dataset results in significant performance gains on benchmarks like VSI-Bench, 3D QA, dense captioning, and object grounding, demonstrating the effectiveness of structured 2D inputs for spatial reasoning.
Key Results
- Zero-shot analysis shows GPT-o3, when provided with BEV images and metadata, achieves over 94% accuracy in relative distance and direction tasks, indicating that well-structured 2D visual cues suffice for complex spatial reasoning.
- Fine-tuning Qwen2.5VL on Struct2D-Set improves benchmark scores by approximately 20%, outperforming baseline models and validating the dataset's quality and method's robustness.
- Ablation studies confirm that incorporating object metadata and rotation alignment significantly boosts task accuracy, emphasizing the importance of prompt design and input structuring for spatial tasks.
Significance
This research addresses the longstanding challenge of enabling large language models to perform spatial reasoning without relying on explicit 3D data. By demonstrating that structured 2D visual inputs—derived from perception modules—can support complex spatial tasks, it opens new avenues for low-cost, scalable applications in robotics, AR/VR, and intelligent scene understanding. The approach reduces dependency on expensive 3D sensors, broadening the deployment potential of spatial reasoning systems and fostering more accessible AI solutions for real-world environments.
Technical Contribution
The paper introduces Struct2D prompting, a novel perception-guided strategy that converts perception outputs into structured 2D inputs for large language models. It constructs a large-scale QA dataset, Struct2D-Set, with automatic generation and augmentation, facilitating multi-task instruction tuning. The approach leverages object detection, BEV rendering, and metadata encoding, combined with carefully designed prompts, to enable models like Qwen2.5VL to perform robust spatial reasoning. This framework bridges perception and language reasoning efficiently, avoiding the need for costly 3D representations, and sets a new standard for multi-task spatial understanding.
Novelty
This work is the first to demonstrate that explicit 3D representations are unnecessary for complex spatial reasoning tasks in large language models. By transforming perception outputs into structured 2D cues—BEV images, object marks, and metadata—it bypasses the need for expensive 3D data or specialized architectures. The combination of perception-guided prompting, large-scale automatic QA generation, and instruction tuning introduces a new paradigm for spatial reasoning, significantly reducing computational costs while maintaining high accuracy. This approach fundamentally shifts how spatial understanding can be integrated into multimodal models.
Limitations
- The method's performance depends heavily on the quality of perception modules; detection errors or occlusions can impair reasoning accuracy, especially in cluttered or dynamic scenes.
- While effective in indoor static environments, its generalization to outdoor or highly dynamic scenarios remains untested and challenging.
- Computational overhead from perception processing and large-scale dataset generation may limit real-time applications, requiring further optimization.
Future Work
Future research will explore integrating dynamic scene understanding, multi-view fusion, and self-supervised learning to enhance robustness. Extending the framework to outdoor environments and real-time applications, as well as incorporating more diverse spatial tasks, will further broaden its applicability. Additionally, developing end-to-end training pipelines that unify perception and reasoning modules could improve accuracy and efficiency.
AI Executive Summary
Spatial reasoning is fundamental for intelligent interaction within 3D environments, yet traditional models rely heavily on explicit 3D data, which is costly and limited in scalability. This paper introduces Struct2D, a perception-guided framework that converts RGB-D videos into structured 2D inputs—bird's-eye view images with object marks and metadata—enabling large language models like GPT-o3 to perform complex spatial reasoning without explicit 3D representations.
The core innovation lies in transforming perception outputs into structured prompts, which include rotated BEV images, filtered object marks, and object-centric metadata. These inputs guide the model through step-by-step reasoning processes, effectively bridging perception and language understanding. Zero-shot analysis demonstrates that GPT-o3, with such structured inputs, achieves over 94% accuracy in relative distance and direction tasks, indicating strong inherent spatial reasoning capabilities.
Building on these insights, the authors develop Struct2D-Set, a large-scale dataset of 200,000 QA pairs covering eight spatial categories, generated automatically from 3D indoor scene datasets like ScanNet and ARKitScenes. Fine-tuning the open-source Qwen2.5VL model on this dataset results in significant performance improvements across benchmarks such as VSI-Bench, dense captioning, and object grounding, validating the approach’s practicality.
Experimental results confirm that carefully designed prompts and structured inputs can unlock robust spatial reasoning in large language models, reducing reliance on expensive 3D data. This work paves the way for scalable, low-cost spatial understanding solutions applicable to robotics, AR/VR, and scene analysis. Future directions include integrating dynamic scene understanding and multi-view perception to further enhance robustness and real-time capabilities.
Deep Analysis
Background
近年来，空间理解在多模态学习中成为研究热点。早期方法如PointNet、PointCNN专注点云处理，提供几何细节，推动了室内场景理解。随后，结合深度学习的3D重建模型如ScanNet、ARKitScenes推动了场景空间感知的发展。多模态大模型（如GPT-4、PaLM-E）在图像和视频理解方面取得突破，但在复杂空间推理中仍受限于输入形式。点云模型虽具几何优势，但成本高、依赖标注，限制了其应用。近年来，利用二维投影（如BEV）进行空间推理成为新趋势，减少了对昂贵3D数据的依赖，但缺乏细粒度空间关系信息。本文在此基础上，提出结构化二维输入，结合感知模块，突破了传统限制，推动空间理解向低成本、高效方向发展。
Core Problem
现有空间推理模型多依赖昂贵的显式3D输入，导致成本高、泛化能力差。多模态模型在理解复杂空间关系、路径规划等任务时表现有限，主要因缺乏有效的二维视觉结构信息。如何在无需昂贵3D重建的情况下，利用二维视觉信息实现准确空间推理，成为亟待解决的问题。这关系到机器人导航、虚拟现实等应用的实用性，也影响多模态模型的普适性和扩展性。现有方法在多任务场景下表现不佳，缺乏统一的空间推理框架，亟需一种低成本、易扩展的解决方案。
Innovation
本研究的核心创新在于提出Struct2D感知引导的二维提示策略，将感知模块输出转化为结构化二维输入，支持多空间推理任务。具体包括：
- �� 利用感知模块提取点云和目标检测，生成BEV图像及元数据，避免昂贵的3D重建。
- �� 设计结构化prompt，结合物体标记、元数据和旋转对齐，显著提升推理准确性。
- �� 构建大规模自动生成的QA数据集，覆盖8类空间任务，支持多任务微调。
- �� 通过微调开源模型Qwen2.5VL，验证其在复杂空间任务中的优越表现。这些创新突破了传统依赖3D输入的限制，为空间推理提供了新思路。
Methodology
- �� 输入RGB-D视频，感知模块提取点云P和目标检测O。
- �� 生成鸟瞰图BEV，标记相关物体，过滤无关对象。
- �� 构建物体元数据Tmeta（类别、坐标等）作为文本输入。
- �� 结合egocentric关键帧Ikeyframe，利用深度投影选择代表性帧。
- �� 构造结构化prompt，包含空间关系推理步骤，指导模型理解。
- �� 使用微调的Qwen2.5VL模型在Struct2D-Set上进行训练，优化多任务性能。
- �� 评估模型在VSI-Bench等空间推理任务中的表现，进行消融分析验证关键组件。
Experiments
采用ScanNet、ARKitScenes等数据集，自动生成20万QA对，涵盖空间关系、路径导航、目标识别等任务。模型在VSI-Bench、3DQA等基准上测试，比较不同prompt设计和感知输入质量的影响。参数调优包括不同的物体过滤策略和旋转对齐方法。通过消融实验验证元数据和结构化prompt的重要性。模型微调采用AdamW优化器，训练时间约为24小时，硬件使用NVIDIA A100 GPU。评估指标包括准确率、F1分数和任务特定的性能指标，确保多任务性能的提升。
Results
微调后，Qwen2.5VL在VSI-Bench的空间推理任务中，平均得分提升20%以上，特别是在相对距离和路径规划中达94.4%和80.1%的准确率。零-shot分析显示，单一BEV图像配合元数据已足够支持复杂推理，验证二维结构信息的有效性。消融实验表明，加入物体元数据和旋转对齐，模型性能提升显著，说明合理prompt设计关键。与未微调模型相比，微调模型在多任务场景中表现更稳健，展现出良好的泛化能力。这些结果证明，结构化二维输入结合感知模块，是实现高效空间推理的可行方案。
Applications
该方法可应用于机器人导航、虚拟现实、智能监控等场景，降低空间理解的硬件成本，提升交互智能水平。只需感知模块提供二维视觉结构，即可实现复杂空间推理，减少对昂贵3D传感器的依赖。未来，结合动态场景和多模态信息，将推动自主系统在复杂环境中的自主导航和交互能力，促进智能空间感知技术的广泛应用。
Limitations & Outlook
目前方法依赖感知模块的检测质量，感知误差可能影响推理效果。在极端遮挡或动态场景中表现有限，模型在未见过的空间布局下泛化能力不足。此外，模型在大规模场景中的计算成本较高，未来需优化感知和推理流程，增强鲁棒性和实时性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，看到桌子、炉子、沙发这些家具。你不用用3D扫描仪，只用眼睛观察，从不同角度看，逐渐在脑海中拼凑出整个厨房的布局。这个过程就像用二维图片和一些简单的标签，帮机器人理解空间关系。比如，知道沙发在炉子旁边，或者门在桌子后面。这样，机器人不用复杂的3D模型，也能理解空间关系，帮你找到东西或规划路线。其实，这就像你用地图和标记，找到从厨房门到冰箱的路径一样。这个方法让机器人变得更聪明、更便宜，也更容易在家里或工厂里用。
ELI14 Explained like you're 14
想象你在玩一个游戏，你的任务是找到房间里的东西，比如沙发、电视和炉子。你不用用特殊的3D扫描器，只用一张平面的地图和一些标记，就能知道这些东西的位置。比如，你知道沙发在炉子旁边，电视在房间的角落。你还可以用这个地图规划走路的路线，找到最快的路径。这就像你用一张平面地图和标记，轻松找到学校里的教室或商场里的店铺一样。科学家们用类似的方法，让机器人也能用二维图片和简单的标签，理解空间关系，不需要复杂的3D模型。这样，机器人就可以更快、更便宜地学会在房间里找到东西，帮你做事情。是不是很酷？
Abstract
Unlocking spatial reasoning in Multimodal Large Language Models (MLLMs) is crucial for enabling intelligent interaction with 3D environments. While prior efforts often rely on explicit 3D inputs or specialized model architectures, we ask: can MLLMs reason about 3D space using only structured 2D representations derived from perception? We introduce Struct2D, a perception-guided prompting framework that combines bird's-eye-view (BEV) images with object marks and object-centric metadata, optionally incorporating egocentric keyframes when needed. Using Struct2D, we conduct an in-depth zero-shot analysis of closed-source MLLMs (e.g., GPT-o3) and find that they exhibit surprisingly strong spatial reasoning abilities when provided with structured 2D inputs, effectively handling tasks such as relative direction estimation and route planning. Building on these insights, we construct Struct2D-Set, a large-scale instruction tuning dataset with 200K fine-grained QA pairs across eight spatial reasoning categories, generated automatically from 3D indoor scenes. We fine-tune an open-source MLLM (Qwen2.5VL) on Struct2D-Set, achieving competitive performance on multiple benchmarks, including 3D question answering, dense captioning, and object grounding. Our approach demonstrates that structured 2D inputs can effectively bridge perception and language reasoning in MLLMs-without requiring explicit 3D representations as input. We will release both our code and dataset to support future research.