PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
PerceptionBench evaluates atomic visual perception in MLLMs; highest accuracy among 16 models is only 59.7%.
Key Findings
Methodology
PerceptionBench uses failure analysis from 42 benchmarks to construct an error taxonomy defining 10 atomic visual perception capabilities. It includes 3,000 verified questions isolating single capabilities.
Key Results
- 16 models tested; GPT-5.6-Sol achieved the highest accuracy at 59.7%, highlighting unsolved atomic perception.
- Perception-related hallucination scored lowest on average, revealing critical weaknesses.
- Models with similar overall scores showed divergent capability profiles, enabling fine-grained diagnostics.
Significance
PerceptionBench provides a granular evaluation standard for visual perception in MLLMs, addressing gaps in existing benchmarks that conflate perception errors with reasoning or knowledge failures.
Technical Contribution
Introduced failure-driven taxonomy defining 10 atomic perception capabilities and constructed a benchmark to comprehensively evaluate these capabilities, enabling detailed diagnostics.
Novelty
First benchmark derived from empirical failures to isolate atomic visual perception capabilities, overcoming the narrow scope of prior benchmarks.
Limitations
- Does not cover all possible perception capabilities; dataset expansion needed.
- Evaluation tied to current frontier models, limiting future applicability.
Future Work
Future directions include expanding benchmark coverage, adding diverse visual scenarios, and improving model perception capabilities.
AI Executive Summary
Multimodal large language models (MLLMs) struggle with visual perception, a foundational skill for downstream reasoning and interaction. Existing benchmarks fail to isolate perception errors from reasoning or knowledge failures.
PerceptionBench addresses this by introducing a failure-driven taxonomy defining 10 atomic visual perception capabilities, such as localization, attribute recognition, and counting. It includes 3,000 verified questions designed to isolate single capabilities, ensuring difficulty stems from perception rather than reasoning.
Results from 16 frontier models reveal atomic perception remains unsolved, with no model surpassing 60% accuracy. Perception-related hallucination scored lowest, highlighting critical weaknesses. PerceptionBench sets a new standard for diagnosing visual perception boundaries and guiding future model improvements.
Deep Analysis
Background
MLLMs have advanced reasoning and knowledge integration but struggle with visual perception. Existing benchmarks conflate perception errors with reasoning failures, limiting diagnostic power.
Core Problem
Atomic visual perception capabilities like localization, counting, and attribute recognition are critical but poorly mastered by current models, restricting their applicability in complex tasks.
Innovation
PerceptionBench introduces a failure-driven taxonomy defining 10 atomic visual perception capabilities and constructs a benchmark isolating these capabilities for fine-grained evaluation.
Methodology
- �� Analyze failures from 42 benchmarks to derive taxonomy.
- �� Define 10 atomic perception capabilities.
- �� Design 3,000 questions isolating single capabilities.
- �� Multi-stage verification ensures accuracy and difficulty balance.
Experiments
16 frontier models, including GPT-5.6-Sol and Claude-Fable-5, were evaluated using unified protocols. Automatic evaluation was cross-validated with human judgments for reliability.
Results
Top accuracy was 59.7% (GPT-5.6-Sol). Perception-related hallucination scored lowest, and models with similar overall scores showed varied capability profiles.
Applications
Useful for diagnosing MLLM weaknesses in visual tasks and guiding improvements in perception-related capabilities.
Limitations & Outlook
Does not cover all perception capabilities; tied to current models. Future work should expand datasets and capabilities.
Plain Language Accessible to non-experts
Think of PerceptionBench as a test system for AI models, like a school exam that focuses on just one skill at a time. For example, one question might ask the model to count objects in a picture, while another asks it to identify colors. By isolating each skill, researchers can pinpoint exactly where the model struggles, just like a teacher identifying a student's weak subjects.
ELI14 Explained like you're 14
Imagine you're playing a cool game with a robot helper that can look at pictures and tell you things like how many cats there are or what color a hat is. But this robot keeps messing up! Scientists made a test called PerceptionBench to see how good these robots are at 'seeing' things. Turns out, the best robot only scored 59.7%! Looks like these robots need a lot more practice before they can ace the test!
Glossary
Atomic Visual Capability
Basic visual skills like counting, localization, and attribute recognition.
Used to define evaluation categories in PerceptionBench.
Perception-related Hallucination
Errors where models generate false visual information.
Scored lowest among tested capabilities.
Error Taxonomy
Classification of error types derived from model failures.
Used to construct PerceptionBench's evaluation framework.
Multimodal Model
AI models capable of processing multiple data types like images and text.
Focus of this study.
Benchmark
Standardized test set for evaluating model performance.
PerceptionBench is designed for visual perception benchmarking.
Open Questions Unanswered questions from this research
- 1 How to improve perception-related hallucination capabilities?
- 2 How to expand benchmarks to cover more visual scenarios?
Applications
Immediate Applications
Model Diagnostics
Helps researchers identify specific weaknesses in models and guide improvements.
Optimizing Visual Tasks
Provides insights for improving MLLM performance in complex visual applications.
Long-term Vision
General Visual Perception Models
Developing models that fully master atomic visual capabilities to advance AI.
Abstract
We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60\% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.