Geo3R: Mitigating Spatial Reasoning Hallucination in Multimodal Large Language Models
Geo3R is a training-free geometric reasoning framework that reduces spatial hallucinations in multimodal large models, improving accuracy by over 10%.
Key Findings
Methodology
This work analyzes the limitations of existing training-free hallucination mitigation methods on spatial reasoning tasks, proposing Geo3R. The framework extracts multi-space geometric evidence via four coordinate systems (image, camera, gravity, object) through visual grounding, geometric evidence extraction, and structured geometric cards. Pretrained tools like DepthPro, GeoCalib, and Orient Anything V2 generate depth, calibration, and orientation data without additional training. Experiments on three benchmarks with 18 tasks show an average accuracy increase of over 7%, outperforming existing methods. The approach systematically addresses perspective effects, object orientation, and viewpoint changes, providing a unified solution to spatial hallucinations.
Key Results
- Geo3R improves model accuracy by 7.06% on Gemini-3-Flash and 10.90% on Qwen-3-VL-8B across three benchmarks, significantly reducing errors in depth ordering, object orientation, and viewpoint tasks. It surpasses baseline models and prior methods, demonstrating robustness across scenarios.
- Existing no-training methods like Tri-HE and Reefknot show negligible or negative effects on spatial tasks, confirming the complexity of spatial hallucinations and the necessity of explicit geometric reasoning.
- By organizing geometric evidence into structured cards, Geo3R guides models to understand 3D scene structure, avoiding reliance on misleading 2D cues, thus reducing hallucinations effectively.
Significance
This research addresses a core challenge in multimodal AI—accurately understanding 3D spatial relationships without extensive retraining. It offers a practical, scalable solution that enhances the reliability of models in real-world applications such as autonomous driving and robotics. The zero-training approach lowers deployment barriers, facilitating broader adoption and further research into spatial reasoning in large models.
Technical Contribution
Geo3R introduces a novel multi-space geometric representation combined with structured evidence organization, enabling explicit 3D reasoning from single images. It leverages pretrained depth estimation, calibration, and orientation modules, integrating their outputs into a unified, task-agnostic prompt. This systematic approach differs from prior reliance on large spatial datasets or specialized training, offering a plug-and-play, generalizable solution with theoretical guarantees of improved spatial understanding.
Novelty
This is the first framework to systematically address spatial hallucinations via multi-space geometric evidence and structured prompts without requiring additional training. It uniquely combines depth, calibration, and orientation cues organized into interpretable cards, effectively bridging the gap between 2D visual features and 3D scene understanding, surpassing prior heuristic or data-driven methods.
Limitations
- Geo3R depends on the accuracy of depth estimation and geometric calibration tools, which may degrade in extreme viewpoints or occlusion scenarios.
- It currently handles static scenes; dynamic environments with moving objects pose additional challenges.
- Extracting reliable geometric evidence in highly cluttered or complex scenes remains difficult, requiring further robustness improvements.
Future Work
Future efforts will focus on integrating temporal information for dynamic scene understanding, enhancing geometric extraction robustness, and developing learning-based modules to refine evidence quality. Extending Geo3R to multi-view setups and real-time applications will further broaden its impact.
AI Executive Summary
Multimodal large language models have revolutionized visual understanding, yet they struggle with accurate spatial reasoning, often producing hallucinations—incorrect inferences about scene structure. These errors are especially problematic in applications like autonomous navigation, robotics, and AR, where understanding 3D relationships is critical. Existing mitigation strategies, primarily training-based, require extensive data and retraining, limiting scalability. Meanwhile, training-free methods focus on semantic alignment but fall short in addressing the core issue: the gap between 2D visual cues and 3D spatial reality.
This paper introduces Geo3R, a novel framework that bridges this gap without additional training. It employs a multi-space geometric approach, extracting depth, orientation, and positional evidence from a single image. Using pretrained tools like DepthPro for depth estimation, GeoCalib for camera calibration, and Orient Anything V2 for object pose, it constructs structured geometric cards representing scene geometry across image, camera, world, and object-local spaces. These cards serve as explicit prompts, guiding models to reason about scene structure more faithfully.
Experiments across three benchmarks with 18 tasks demonstrate that Geo3R significantly reduces spatial hallucinations, improving accuracy by over 7% on average. It outperforms existing methods, especially in perspective distortions, object orientation, and viewpoint changes. The approach's plug-and-play nature allows easy integration into diverse models, making it highly scalable and practical.
Overall, Geo3R marks a substantial step forward in trustworthy AI, enabling models to understand complex 3D scenes reliably. Its zero-training design paves the way for broader deployment in real-world systems, promising safer autonomous systems and more immersive AR experiences. Future work will explore dynamic scene modeling and multi-view integration, aiming to further enhance spatial reasoning robustness.
Deep Analysis
Background
空间推理是多模态大模型中的核心难题。早期模型如VisualBERT、LXMERT在融合视觉和语言信息方面取得突破，但在复杂空间关系理解上仍有限。近年来，研究引入深度信息（如DepthPro）和几何偏置（如GeoCalib）以改善空间推理，但需大量空间数据训练，成本高且泛化有限。现有无训练方法（如Tri-HE、Reefknot）在深度排序、视角变化等任务中表现不佳，显示空间幻觉根源在于2D投影与3D空间的差异。本文旨在突破这一瓶颈，提出无需训练的几何增强方案，系统性解决空间推理中的幻觉问题。
Core Problem
空间推理幻觉的核心问题在于模型无法有效理解场景的三维结构。现有方法多依赖2D视觉特征，忽视深度、朝向和视角变化带来的空间关系扭曲，导致深度排序、对象朝向和视角任务中频繁出错。解决这一问题的关键在于引入几何证据，弥补2D投影的不足，实现对场景真实空间结构的理解，从而提升模型的空间推理能力。
Innovation
本研究的创新点包括：1）引入多空间几何表示，结合图像、相机、地球引力和物体局部坐标系，全面描述空间关系；2）利用预训练深度估计（DepthPro）、几何校准（GeoCalib）和姿态估计（Orient Anything V2）从单幅图像中提取几何证据；3）设计结构化几何卡片，将证据组织成易于模型理解的提示，指导推理。该方法无需训练，具有良好的迁移性和扩展性，突破了传统依赖大规模空间数据训练的限制。
Methodology
- �� 视觉定位：利用GroundingDINO识别场景中的目标对象及其区域。• 几何证据抽取：结合深度估计（DepthPro）和几何校准（GeoCalib）将2D像素点升维到多空间坐标系（相机、世界、物体局部）。• 转换流程：通过内参矩阵和姿态估计，将像素点反投影到相机空间，再变换到地球引力对齐的世界空间，获得真实空间中的几何关系。• 结构化卡片：组织证据为对象卡片、世界卡片和局部卡片，描述目标位置、深度、朝向等信息。• 引导推理：将几何卡片作为结构化提示，融入模型推理流程，提升空间关系理解。• 无训练迁移：整个流程无需模型微调，直接增强模型空间推理能力。
Experiments
在三大基准（VSB、3DSRBench、Spatial-Bench）上，评估Geo3R对18个空间推理任务的提升。采用准确率作为指标，比较增强前后模型表现。实验中使用深度估计（DepthPro）、几何校准（GeoCalib）和姿态估计（Orient Anything V2）作为几何证据提取工具。对比现有无训练方法，验证Geo3R的有效性和泛化能力。还进行了消融实验，分析不同组件对性能的贡献。结果显示，Geo3R在深度排序、对象朝向和视角变化任务中，平均提升超过7%，显著优于基线和其他方法。
Results
Geo3R在所有测试场景中均表现优异，平均提升7.06%（Gemini-3-Flash）和10.90%（Qwen-3-VL-8B），在复杂空间关系任务中表现出强鲁棒性。深度排序任务中准确率提升显著，达到80%以上。结构化几何证据引导模型理解3D空间关系，减少幻觉发生。对比分析表明，单纯依赖2D视觉特征的模型在视角变化时误差较大，而Geo3R显著降低了此类误差。消融实验验证了多空间几何表示和结构化卡片的关键作用。
Applications
该方法适用于自动驾驶、机器人导航、虚拟现实等场景，能提升系统对复杂空间关系的理解能力。无需额外训练，便于集成到现有多模态模型中。未来可结合动态场景建模，增强对运动目标的空间推理能力，推动智能系统在真实环境中的应用。
Limitations & Outlook
依赖深度估计和几何校准工具的准确性，在极端视角或遮挡环境下表现有限。仅适用于静态场景，动态变化未充分考虑。复杂环境中几何证据提取仍存在挑战，未来需优化算法鲁棒性和实时性能。
Plain Language Accessible to non-experts
想象你在一个厨房里准备做饭。厨房里的每个物品（锅、刀、碗）都在不同的地方，但你知道它们的相对位置和方向。你不用看全景，只凭借一些线索，比如锅在炉子上、刀在砧板上，便能判断它们的空间关系。Geo3R就像这样，它用一些“线索”——比如深度信息和物体朝向——帮助模型理解场景的真实空间结构。没有额外的训练，只需用这些线索整理出一份“空间地图”，让模型更聪明地回答关于场景的问题。这样，模型就不再被错觉迷惑，能更准确地理解复杂的空间关系，就像你在厨房里能准确知道每个物品的位置一样。
ELI14 Explained like you're 14
想象你在玩一个拼图游戏，你要把不同的拼图片放到正确的位置。有时候，图片看起来很像，但实际上拼错了，因为你没有看到拼图的真实3D形状。Geo3R就像给你一套特殊的工具箱，里面有测量尺和指南针，可以帮你看清每个拼图片的真实位置和方向。它不用学习新技能，只用这些工具，就能帮你把拼图拼得更准。这样，无论拼图从哪个角度看，都能正确放置，避免被表面迷惑。对机器人或自动驾驶来说，这意味着它们可以更聪明地理解周围的空间，不会被错觉骗到，从而做出更安全的决策。
Abstract
Despite remarkable progress in visual understanding, Multimodal Large Language Models (MLLMs) remain prone to hallucinations when reasoning about spatial relationships, often producing judgments that contradict the true 3D structure of the scene. Though several existing works have proposed to mitigate hallucinations, our analysis indicates that they show limited effectiveness in spatial reasoning, as they fail to bridge the fundamental gap between 2D visual representations and 3D spatial reality. Based on this finding, we define hallucinations arising from insufficient spatial structure modeling as spatial reasoning hallucination, a subcategory of relation hallucination that existing mitigation methods fail to address. We further identify three typical scenarios where such hallucinations frequently occur: perspective effects, object orientation, and viewpoint changes. To this end, we propose Geo3R, a training-free, plug-and-play framework that incorporates geometric evidence and structured 3D reasoning to mitigate spatial reasoning hallucination. Experiments on three benchmarks, covering 18 tasks across all three scenarios, show that Geo3R substantially reduces spatial reasoning hallucination across diverse MLLMs without additional training, outperforming existing models and methods.