OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models
OddGridBench reveals MLLMs' deficiency in visual discrepancy detection, proposing OddGrid-GRPO to enhance performance.
Key Findings
Methodology
The paper introduces OddGridBench, a benchmark with over 1,400 grid images to evaluate MLLMs' visual discrepancy sensitivity. Each image has one element differing in color, size, rotation, or position. OddGrid-GRPO, a reinforcement learning framework, is proposed to enhance visual discrimination through curriculum learning and distance-aware reward.
Key Results
- Experiments show all evaluated MLLMs perform far below human levels in visual discrepancy detection. Qwen3-VL-32B achieved the highest overall accuracy at 68.07%.
- OddGrid-GRPO significantly improved model performance in visual discrepancy detection, reaching an accuracy of 70.86%.
- Human participants achieved a total accuracy of 87.47%, significantly outperforming models.
Significance
The study reveals current MLLMs' deficiencies in low-level visual perception, particularly in detecting fine-grained discrepancies. By introducing new benchmarks and reinforcement learning frameworks, the paper lays the groundwork for perceptual grounding in multimodal intelligence, aiding complex visual tasks.
Technical Contribution
The paper presents OddGridBench, a controllable benchmark for systematically evaluating MLLMs' visual discrepancy sensitivity. OddGrid-GRPO combines curriculum learning and distance-aware rewards to significantly enhance fine-grained visual discrimination.
Novelty
OddGridBench is the first benchmark focused on evaluating MLLMs' visual discrepancy sensitivity. OddGrid-GRPO offers a novel solution by combining curriculum learning and distance-aware rewards to enhance visual perception.
Limitations
- Current models perform poorly in rotation and position discrepancy detection, possibly due to the complexity of these attributes and insufficient perceptual capabilities.
- Training requires substantial computational resources, potentially limiting application in resource-constrained environments.
- Performance in multi-attribute combinations still has room for improvement.
Future Work
Future work could explore more complex visual scenes and visual discrepancy detection in multimodal tasks. Integrating other machine learning methods, such as self-supervised learning, may further enhance perceptual capabilities.
AI Executive Summary
Multimodal large language models (MLLMs) have excelled in vision-language tasks but still struggle with low-level visual perception, particularly in detecting fine-grained discrepancies. OddGridBench systematically evaluates MLLMs' visual discrepancy sensitivity through over 1,400 grid images, each featuring one element differing in color, size, rotation, or position. Experiments reveal all evaluated MLLMs perform far below human levels in visual discrepancy detection. To address this, OddGrid-GRPO, a reinforcement learning framework, is proposed to enhance visual discrimination through curriculum learning and distance-aware rewards. OddGrid-GRPO significantly improves fine-grained visual discrimination, achieving an accuracy of 70.86%. The study highlights current MLLMs' deficiencies in low-level visual perception, particularly in detecting fine-grained discrepancies. By introducing new benchmarks and reinforcement learning frameworks, the paper lays the groundwork for perceptual grounding in multimodal intelligence, aiding complex visual tasks.
Deep Analysis
Background
Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in vision-language tasks, mainly focusing on high-level visual understanding and reasoning. However, existing evaluations often overlook an essential aspect of human vision: the ability to detect fine-grained discrepancies in visual scenes. This perceptual sensitivity is crucial for robust spatial reasoning, object understanding, and visual question answering.
Core Problem
Despite MLLMs' success in high-level visual tasks, they still struggle with low-level visual perception, particularly in detecting fine-grained discrepancies. Existing benchmarks often overlook this critical aspect, leading to models' inadequate performance in complex visual tasks.
Innovation
The paper introduces OddGridBench, a controllable and scalable benchmark designed to systematically evaluate MLLMs' visual perceptual discrepancy sensitivity. By generating grid-based images, OddGridBench enables quantitative and systematic analysis across multiple perceptual dimensions. Additionally, OddGrid-GRPO, a reinforcement learning framework combining curriculum learning and distance-aware rewards, significantly enhances models' fine-grained visual discrimination.
Methodology
- �� OddGridBench generates over 1,400 grid images, each featuring one element differing in color, size, rotation, or position.
- �� OddGrid-GRPO uses curriculum learning to progressively control training sample difficulty and incorporates spatial proximity constraints into reward design.
- �� Distance-aware rewards provide continuous perceptual feedback, enhancing visual discrepancy detection.
Experiments
Experiments were conducted on the OddGridBench benchmark, evaluating 19 representative MLLMs, including open-source and proprietary systems. Using the AdamW optimizer, learning rate was set to 1×10−6, weight decay to 1×10−2, gradient clipping at 1.0. All experiments used identical configurations for fair comparison.
Results
Experiments show all evaluated MLLMs perform far below human levels in visual discrepancy detection. Qwen3-VL-32B achieved the highest overall accuracy at 68.07%. OddGrid-GRPO significantly improved model performance in visual discrepancy detection, reaching an accuracy of 70.86%.
Applications
OddGridBench and OddGrid-GRPO lay the groundwork for perceptual grounding in multimodal intelligence, aiding complex visual tasks. Applications include autonomous driving, robotic vision, and augmented reality.
Limitations & Outlook
Current models perform poorly in rotation and position discrepancy detection, possibly due to the complexity of these attributes and insufficient perceptual capabilities. Training requires substantial computational resources, potentially limiting application in resource-constrained environments. Performance in multi-attribute combinations still has room for improvement.
Plain Language Accessible to non-experts
Imagine you are shopping in a large supermarket with shelves full of various products. You need to find a product that is slightly different from the others, such as a slightly different color or a slightly shifted position. OddGridBench is like this supermarket, evaluating models' visual discrepancy sensitivity by generating grid images. OddGrid-GRPO is a smart shopping assistant that helps you find the different product faster by continuously learning and adjusting.
ELI14 Explained like you're 14
Hey, kids! Imagine you're playing a spot-the-difference game with lots of similar patterns on the screen, but one is a bit different. This game is like OddGridBench, helping us evaluate computer models' ability to spot differences. OddGrid-GRPO is like a super helper, practicing to make the model better at spotting differences. Isn't that cool?
Glossary
Multimodal Large Language Models
Models that combine visual and language information to handle complex vision-language tasks.
Used to evaluate visual discrepancy sensitivity.
Visual Discrepancy
Subtle differences between elements in a visual scene, such as color, size, rotation, or position.
OddGridBench evaluates models' ability to detect visual discrepancies.
Reinforcement Learning
Training models through reward mechanisms to improve performance in specific tasks.
OddGrid-GRPO uses reinforcement learning to enhance visual discrimination.
Curriculum Learning
Gradually increasing training sample difficulty to stabilize the learning process.
OddGrid-GRPO improves models' visual sensitivity through curriculum learning.
Distance-aware Reward
Provides continuous feedback based on spatial proximity between predicted and ground-truth positions.
OddGrid-GRPO uses distance-aware rewards to enhance visual discrepancy detection.
Open Questions Unanswered questions from this research
- 1 How to improve MLLMs' fine-grained discrepancy detection in complex visual scenes remains an open question.
- 2 Existing methods still have room for improvement in multi-attribute combinations, requiring further research.
Applications
Immediate Applications
Autonomous Driving
OddGridBench and OddGrid-GRPO can enhance autonomous driving systems' visual perception capabilities, helping identify subtle changes on the road.
Long-term Vision
Augmented Reality
Improved visual discrepancy detection can enable augmented reality applications to interact more naturally with the real world, achieving a more seamless user experience.
Abstract
Multimodal large language models (MLLMs) have achieved remarkable performance across a wide range of vision language tasks. However, their ability in low-level visual perception, particularly in detecting fine-grained visual discrepancies, remains underexplored and lacks systematic analysis. In this work, we introduce OddGridBench, a controllable benchmark for evaluating the visual discrepancy sensitivity of MLLMs. OddGridBench comprises over 1,400 grid-based images, where a single element differs from all others by one or multiple visual attributes such as color, size, rotation, or position. Experiments reveal that all evaluated MLLMs, including open-source families such as Qwen3-VL and InternVL3.5, and proprietary systems like Gemini-2.5-Pro and GPT-5, perform far below human levels in visual discrepancy detection. We further propose OddGrid-GRPO, a reinforcement learning framework that integrates curriculum learning and distance-aware reward. By progressively controlling the difficulty of training samples and incorporating spatial proximity constraints into the reward design, OddGrid-GRPO significantly enhances the model's fine-grained visual discrimination ability. We hope OddGridBench and OddGrid-GRPO will lay the groundwork for advancing perceptual grounding and visual discrepancy sensitivity in multimodal intelligence. Code and dataset are available at https://wwwtttjjj.github.io/OddGridBench/.