SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
Proposed SSMNBench, a diagnostic benchmark with 3300 QA pairs, distinguishes single-view sufficiency and multi-view necessity to evaluate cross-view understanding of 17 models.
Key Findings
Methodology
The study constructs SSMNBench with 3300 QA pairs across 11 tasks, categorizing them into SVS and MVN. By systematically perturbing view availability in 17 state-of-the-art models, it analyzes performance drops and geometric fusion failures. The benchmark employs expert annotation, view randomization, distractor generation, and introduces the Distraction Decay metric to quantify robustness against redundant views. This comprehensive setup ensures precise diagnosis of models’ spatial reasoning capabilities.
Key Results
- Experiments reveal that most models suffer performance degradation exceeding 15% when redundant views are added, indicating reliance on semantic averaging rather than true geometric fusion. Proprietary models like GPT-5.2 reach only 47.5% accuracy in SVS tasks, far below human performance of over 87%. Multi-view inputs sometimes introduce interference, exposing models’ fragility in integrating fragmented geometric cues, especially in occlusion-heavy scenarios.
- Models tend to depend on semantic priors and view preferences, lacking deep 3D spatial understanding. The perturbation experiments show significant performance declines in MVN tasks, emphasizing the importance of geometric reasoning. The benchmark’s diagnostic results highlight the need for models to improve in spatial perception and robustness.
- This work provides a clear evaluation framework for future development, emphasizing the necessity of spatial geometry comprehension and anti-interference mechanisms to achieve human-like multi-view understanding in complex environments.
Significance
This research exposes fundamental limitations of current multimodal models in complex human-centric scenes, especially in multi-view geometric reasoning. By providing a systematic diagnostic framework, it guides the development of models capable of true spatial understanding, which is crucial for applications like surveillance, virtual reality, and autonomous navigation. The benchmark addresses a critical gap in evaluating multi-view reasoning, fostering progress toward models that can reliably interpret multi-camera data in real-world scenarios, thus bridging the gap between AI perception and human visual cognition.
Technical Contribution
The study introduces a novel evaluation taxonomy distinguishing SVS and MVN, along with a high-quality multi-task dataset covering dense occlusion-heavy scenes. It combines view perturbation, distractor generation, and the Distraction Decay metric to diagnose geometric fusion deficiencies. This approach moves beyond traditional accuracy metrics, emphasizing the importance of spatial reasoning and robustness. The framework enables precise identification of models’ weaknesses in multi-view geometric integration, guiding targeted improvements.
Novelty
This is the first work to systematically differentiate single-view sufficiency from multi-view necessity in a diagnostic benchmark for multimodal models. It innovatively employs view perturbation and distractor strategies, along with the Distraction Decay metric, to quantify models’ spatial reasoning robustness. Unlike prior benchmarks that treat multi-view inputs as a fixed set, SSMNBench explicitly evaluates the models’ ability to selectively attend and fuse fragmented geometric cues, marking a significant advancement in understanding multi-view perception.
Limitations
- The benchmark primarily focuses on static scenes; dynamic multi-view video understanding remains unaddressed. Extending to temporal sequences is necessary for real-time applications.
- Models still struggle under extreme occlusion and complex interactions, indicating that spatial reasoning depth needs further enhancement.
- Current experiments involve pre-trained vision-language models; pure geometric models or those with explicit 3D reasoning modules are less explored, representing future research directions.
Future Work
Future efforts will integrate 3D scene reconstruction and temporal modeling to improve dynamic scene understanding. Developing models with explicit geometric reasoning modules and multi-view attention mechanisms could further enhance robustness. Additionally, expanding the benchmark to include real-time video sequences and more diverse environments will be crucial for deploying these models in practical applications.
AI Executive Summary
Multimodal large language models (MLLMs) have made significant strides in single-image perception, yet their capacity for complex cross-view human-centric scene understanding remains underexplored. Existing benchmarks often evaluate models using a fixed set of views, conflating robustness to visual distraction with genuine multi-view fusion ability. To address this gap, Guo et al. introduce SSMNBench, a diagnostic benchmark comprising 3300 curated question-answer pairs across 11 tasks, designed explicitly to differentiate between single-view sufficiency (SVS) and multi-view necessity (MVN). The benchmark employs systematic view perturbation, expert annotation, distractor generation, and introduces the Distraction Decay metric to quantify a model’s ability to selectively attend and fuse fragmented geometric cues. Experiments across 17 state-of-the-art models reveal that most models suffer performance drops exceeding 15% when redundant views are added, exposing their reliance on semantic averaging rather than true spatial reasoning. Proprietary models like GPT-5.2 achieve only about 47.5% accuracy in SVS tasks, far below human performance of over 87%, highlighting the persistent gap. The findings underscore the importance of developing models with robust spatial understanding and anti-interference capabilities. This work advances the evaluation paradigm for multi-view understanding, providing a clear pathway for future research to enhance models’ 3D perception and reasoning in complex environments. Ultimately, SSMNBench offers a vital tool for pushing the boundaries of AI’s human-like scene comprehension in multi-camera systems.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Multimodal Large Language Models (MLLMs) have shown remarkable progress in single-image perception, yet their ability to reason about complex cross-view human-centric scenes remains largely unverified. Current multi-view benchmarks evaluate models using a fixed "bag of frames" and thus conflate a model's robustness to visual distraction with its genuine ability to fuse fragmented cross-view evidence. To address this issue, we introduce SSMNBench, a diagnostic benchmark comprising 3,300 curated QA pairs for cross-view human and human-object understanding. SSMNBench uniquely categorizes tasks into Single-View Sufficiency (SVS) and Multi-View Necessity (MVN). By systematically perturbing view availability across 17 state-of-the-art MLLMs, critical limitations are revealed: models suffer from severe "distraction degradation" when presented with redundant views (SVS), and fail to integrate fragmented geometric evidence across cameras (MVN). Our evaluations demonstrate that modern MLLMs rely on multiple single-image semantic averaging and view preference rather than genuine cross-view synthesis. By exposing these fundamental vulnerabilities, SSMNBench provides a rigorous diagnostic framework to drive the advancement of future cross-view-aware multimodal architectures. The code is available at: $ \href{https://github.com/gtc-gh/SSMNBench}{\text{SSMNBench}} $