DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues
DiCoBench introduces a high-resolution multi-image benchmark focusing on differential and commonality visual cues, revealing models' significant gaps in fine-grained perception, with top accuracy at 58.1%.
Key Findings
Methodology
DiCoBench employs images approaching 2K resolution, structured into 8 fine-grained tasks divided into differential and commonality cues. The dataset of 765 samples is generated via automated micro-editing with GPT-5.1 and FLUX. The benchmark formulates tasks as multiple-choice questions, including options for no difference or no commonality, to eliminate bias. Evaluation across 18 diverse models shows a stark performance gap, with the best model Gemini-3-Pro achieving only 58.1%, far below human accuracy of 98.3%. This setup rigorously tests models' ability to perceive minute visual details without explicit textual guidance.
Key Results
- Models perform poorly on micro-scale attribute, entity, spatial, and reasoning tasks, with the highest accuracy at 58.1%, highlighting a significant deficiency in autonomous high-resolution perception. The performance varies greatly across tasks, especially in logical reasoning, indicating models' limited capacity to actively perceive subtle visual cues.
- Across all tasks, models lag behind human performance by over 40%, with notable struggles in recognizing fine attribute changes, spatial relationships, and complex reasoning. The results demonstrate that current models lack the robustness needed for detailed visual understanding in high-resolution multi-image scenarios.
- Analysis reveals that models excel in categorical tasks but falter in high-level reasoning and spatial grounding, emphasizing the need for architectures capable of detailed, autonomous perception of micro-level cues in complex environments.
Significance
This work establishes a rigorous high-resolution, multi-task benchmark that exposes the limitations of current multimodal large models in fine-grained, autonomous perception. Moving beyond traditional low-res, text-guided evaluations, DiCoBench provides a systematic framework to measure models’ ability to perceive subtle visual cues, which is critical for real-world applications like autonomous driving, robotics, and scene understanding. It drives the community toward developing models capable of proactive, detailed visual comprehension, aligning AI closer to human perceptual capabilities.
Technical Contribution
The paper introduces a novel benchmark combining high-resolution imagery, multi-task evaluation, and bias-free multiple-choice questions. It leverages automated data synthesis with GPT-5.1 and FLUX for micro-editing, ensuring high-quality samples. The multi-task design covers attributes, entities, spatial relations, and reasoning, providing a comprehensive assessment of models’ fine-grained perception. The evaluation of 18 models highlights the gap in current architectures, guiding future innovations in autonomous visual understanding.
Novelty
This is the first benchmark explicitly targeting high-resolution, multi-image fine-grained perception through differential and commonality cues. Its multi-task, bias-free MCQ format distinguishes it from prior low-res, text-dependent datasets like V* or IDC. The focus on micro-scale details and implicit cues aligns more closely with human perception, marking a significant step forward in autonomous visual cognition research.
Limitations
- The benchmark currently focuses on static images, lacking dynamic or video-based perception scenarios, which are vital for real-time applications.
- Model evaluation is constrained by existing architectures; incorporating advanced reasoning and multi-modal fusion remains a challenge.
- Sample diversity, while extensive, can be expanded further to include more complex backgrounds and micro-detail scenarios for broader generalization.
Future Work
Future efforts will extend to dynamic scene understanding, integrating temporal and causal reasoning. Enhancing data diversity and scale, along with developing architectures that better fuse high-resolution visual cues, will be prioritized. Additionally, exploring real-time perception in videos and deploying models in practical environments like robotics and autonomous vehicles are promising directions.
AI Executive Summary
Recent advances in high-resolution visual encoders and large-scale vision-language datasets have propelled multimodal large language models (MLLMs) to excel in single-image fine-grained perception tasks. Benchmarks such as V* and HR-Bench primarily evaluate models' passive ability to localize and recognize minute details based on explicit textual prompts. However, in real-world environments, perception is often driven by spontaneous, implicit visual cues—subtle differences or commonalities across multiple observations—that humans actively seek without explicit guidance.
Despite progress, current models struggle significantly in autonomous perception of high-resolution, multi-image scenes. To address this gap, the authors introduce DiCoBench, a comprehensive benchmark designed to evaluate models' ability to perceive differential and commonality cues in high-res images. The dataset comprises 765 meticulously curated samples with images approaching 2K resolution, covering 8 tasks including attribute change, entity substitution, spatial relationships, and reasoning. These tasks are formulated as multiple-choice questions, with options for no difference or no commonality, effectively removing bias from text-based evaluation.
Evaluation results across 18 models reveal a stark performance gap: the top model, Gemini-3-Pro, achieves only 58.1% accuracy, far below human performance at 98.3%. Models perform relatively better on categorical tasks but show severe deficiencies in high-level logical reasoning and spatial grounding, especially when micro-scale details are involved. These findings highlight the current limitations of multimodal models in autonomous, high-resolution perception, emphasizing the need for architectures capable of active, micro-level visual understanding.
This work marks a significant step toward more human-like visual cognition in AI systems. By systematically challenging models with high-resolution, multi-task perception, DiCoBench sets a new standard for evaluating and advancing autonomous visual perception. Future research will focus on extending the benchmark to dynamic scenes, improving reasoning capabilities, and deploying models in real-world applications such as robotics and autonomous driving, ultimately bridging the gap between AI and human perceptual abilities.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive fine-grained perception capabilities. However, existing benchmarks predominantly rely on explicit textual cues or low-resolution inputs, failing to evaluate a model's ability to autonomously perceive implicit visual cues in high-resolution. To bridge this gap, we introduce DiCoBench, a comprehensive, multi-image high-resolution benchmark designed for cross-image fine-grained perception. DiCoBench consists of 765 meticulously curated samples categorized into two progressive tracks: Differential Visual Cues and Commonality Visual Cues, covering 8 distinct perception tasks. By formulating the benchmark as a multiple-choice question task and utilizing high-resolution imagery (approaching 2K), we eliminate evaluation metric bias and pose a substantial challenge to current state-of-the-art MLLMs. Our extensive evaluation of 18 diverse MLLMs reveals a striking performance gap compared to human accuracy (98.3\%), with top-performing models struggling significantly with micro-scale detail capture. We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.