MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs
MultiView-Bench evaluates VLM multi-view spatial reasoning; models struggle with 3D relations, improved by active viewpoint fusion (+12-20%).
Key Findings
Methodology
This study introduces MultiView-Bench, a diagnostic benchmark that employs fixed global coordinate systems and procedurally generated multi-view scenes with geometric and real-world assets. It evaluates 15 state-of-the-art VLMs across five task variants and bias analysis tasks. The framework incorporates active viewpoint selection and evidence fusion via the ViewNavigator system, aiming to diagnose and enhance multi-view spatial reasoning capabilities.
Key Results
- In complex 3D relation tasks (DoF=3), the best model GPT-5.6 Sol achieves only 63.3% accuracy, far below deployment standards. Multi-view evidence significantly boosts performance, with an average gain of 25.5 percentage points, especially benefiting newer models. Older models show limited improvement, indicating a gap in multi-view integration.
- Models perform well on 2D relations but poorly on 3D spatial coordinate recognition, with biases toward standard axes and color schemes. Newer models better utilize multiple views, gaining around 25% accuracy with six views, but still struggle with world-centric 3D reasoning.
- ViewNavigator, combining active view selection and evidence aggregation, improves model accuracy by 12.3-20% under fixed view budgets, demonstrating its effectiveness for complex spatial tasks.
Significance
This work highlights critical limitations of current VLMs in understanding 3D spatial relations and integrating multi-view information, essential for robotics, CAD, and assembly tasks. The benchmark provides a systematic tool for diagnosing these shortcomings, guiding future model development. The demonstrated improvements via active view strategies pave the way for more robust spatial reasoning systems, addressing long-standing challenges in AI perception and cognition.
Technical Contribution
The paper introduces MultiView-Bench, a comprehensive diagnostic platform combining procedural scene generation, bias analysis, and active view-based evidence fusion. The ViewNavigator framework exemplifies how integrating active perception with probabilistic reasoning can substantially enhance multi-view spatial understanding, setting a new standard for evaluating and improving VLMs' 3D reasoning capabilities.
Novelty
This is the first benchmark explicitly targeting world-centric multi-view spatial reasoning in VLMs, contrasting with existing datasets focused on pixel-level or egocentric tasks. The integration of active viewpoint selection and bias analysis provides novel insights into model limitations and biases, offering a new direction for research in multi-view 3D understanding.
Limitations
- Models still underperform on complex non-convex geometries and real-world scenes, indicating room for improvement in generalization. Biases toward standard axes and color schemes suggest reliance on priors rather than true spatial understanding.
- The benchmark relies on synthetic and limited real-world assets, which may not fully capture scene diversity. Computational costs for active view strategies are high, limiting real-time applications.
- Future work should focus on scaling scene complexity, reducing computational overhead, and improving robustness to diverse scene variations.
Future Work
Future directions include expanding scene diversity to more complex, real-world scenarios, integrating self-supervised learning to reduce biases, and optimizing active perception algorithms for efficiency. Extending the benchmark to include dynamic scenes and real-time reasoning tasks will further bridge the gap between laboratory performance and practical deployment, advancing AI spatial cognition.
AI Executive Summary
Despite rapid advancements in vision-language models (VLMs), their ability to understand and reason about 3D scenes from multiple viewpoints remains limited. Existing benchmarks primarily evaluate pixel-level mapping or camera-relative navigation, leaving a significant gap in assessing models' capacity for world-centric spatial reasoning. Recognizing this, the authors introduce MultiView-Bench, a novel diagnostic benchmark designed to evaluate how well models can integrate observations from multiple views into a coherent 3D understanding grounded in a fixed global coordinate system.
The benchmark employs procedurally generated scenes with geometric primitives and real-world assets, rendered from multiple viewpoints, and annotated with precise spatial relations. It features five task variants with increasing complexity, from simple 2D relations to full 3D spatial reasoning (DoF=3). Evaluation of 15 leading VLMs reveals that even the most advanced models perform poorly on complex 3D tasks, with accuracy below 65%. Models excel at 2D planar relations but falter when identifying axes and translating multi-view evidence into global spatial relations. Notably, biases towards standard axes and color schemes further impair performance.
To address these challenges, the authors propose ViewNavigator, an active perception framework that employs probabilistic belief updating and strategic viewpoint selection. This approach significantly improves model performance, with gains of up to 20 percentage points under fixed view budgets. The findings underscore the importance of active perception and evidence fusion in advancing AI spatial reasoning.
Overall, this work provides a crucial diagnostic tool for understanding current limitations and guiding future research. It emphasizes the need for models to develop robust, world-centric 3D understanding, which is vital for applications in robotics, CAD, and automated assembly. The benchmark and proposed methods mark a step forward in closing the gap between current AI perception and real-world spatial cognition, paving the way for more intelligent, spatially aware systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在玩拼图游戏，但每次你只能看到一部分拼图，而且角度不同。你需要不断转动拼图，观察不同的面，才能拼出完整的图像。现在，假设你有一个聪明的助手，它可以主动告诉你下一步应该看哪个角度，帮你更快拼好。这个研究就是在教电脑如何像人一样，从多个角度观察场景，理解每个部分在整体中的位置。科学家发现，虽然电脑可以认出单个物体，但在理解它们如何组成完整的3D场景时，还存在很多困难。通过设计专门的测试和智能助手，帮助电脑学会用多角度观察和推理，未来可以让机器人更好地装配零件，或者让3D设计变得更智能、更方便。
ELI14 Explained like you're 14
你知道吗？当你拼乐高积木时，你会从不同角度观察每一块，然后想象它们怎么组合成一座房子。电脑也在学这个技能，但它还不太聪明。有时候，它能认出一块积木，但不知道它在整个房子里的正确位置，尤其是在只看一面的时候。这个研究就像给电脑装上了一个“多角度观察器”，让它可以主动选择看哪个角度，收集更多线索，拼出完整的场景。科学家们设计了一个特别的测试，让电脑在虚拟的场景中练习空间感知。结果发现，虽然新一代的模型比以前更聪明，但在理解复杂空间关系时仍有困难。未来，这项技术可以帮助机器人更好地装配零件，或者让3D设计变得更智能、更方便。
Abstract
Recent benchmarks for VLMs largely assess single- or limited-view perception, leaving untested the core cognitive ability to integrate observations across viewpoints into a coherent, world-centric (allocentric) 3D mental model. We introduce MultiView-Bench, a diagnostic benchmark expressly designed to evaluate multi-view integration for holistic 3D scene comprehension. Unlike existing datasets that focus on pixel-level mapping or camera-relative navigation, MultiView-Bench requires models to decouple object positioning from transient perspectives and ground them in a fixed global coordinate system. This capability serves as a prerequisite for VLMs before being deployed for downstream tasks such as mechanical part assembly. Our systematic evaluation of frontier VLMs reveals consistent failure modes: strong performance on 2D planar relations from a single image, but marked difficulty with 3D spatial relations and with aggregating information across views. We further identify biases in VLMs, such as struggles with unconventional axis directions and sensitivity to object colorways and texture variations. Acknowledging these limitations, we propose ViewNavigator, which uses active viewpoint selection and evidence fusion to improve four base models by 12.3--20.0 percentage points under a six-image cap matching the fixed-view baseline; budget-extended gains are model-dependent and reach 27 percentage points for GPT-5.