Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
The study reveals limitations in VLMs' spatial reasoning for non-textual elements, with text-recognition pathways outperforming visual pathways.
Key Findings
Methodology
The study generates 15 binary 15x15 grids rendered as text symbols and filled squares, testing three frontier VLMs (Claude Opus, ChatGPT 5.2, Gemini 3 Thinking) for transcription ability. It examines differences in spatial reasoning performance between text-recognition and visual pathways.
Key Results
- In the text-symbol condition, Claude and ChatGPT achieve approximately 91% cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all models' accuracy drops to 60-73% and F1 to 29-39%.
- The F1 gap between text and squares ranges from 34 to 54 points, indicating that text-recognition pathways significantly outperform visual pathways in spatial reasoning.
- Each model exhibits distinct failure modes in the squares condition: systematic under-counting (Claude), massive over-counting (ChatGPT), and template hallucination (Gemini).
Significance
The study highlights the inadequacy of current VLMs in spatial localization of non-textual visual elements, challenging the assumption of robust spatial reasoning over arbitrary visual inputs. It underscores the importance of text-recognition pathways in spatial reasoning.
Technical Contribution
The study systematically reveals the performance differences between text-recognition and visual pathways in VLMs, providing experimental evidence for this phenomenon's universality. It offers new perspectives and ideas for improving VLMs' spatial reasoning capabilities.
Novelty
This study is the first to use simple binary grid experiments to reveal performance differences between text-recognition and visual pathways in VLMs, offering new insights into models' spatial reasoning capabilities.
Limitations
- The study is limited to 15x15 binary grids, which may not fully reflect models' performance in complex scenarios.
- Experiments are conducted on three specific VLMs, which may not apply to other models.
Future Work
Future research can extend to more complex visual scenarios, explore the impact of different model architectures on text-recognition and visual pathway performance, and develop new methods to enhance visual pathway spatial reasoning capabilities.
AI Executive Summary
Vision-language models (VLMs) have shown impressive capabilities in visual understanding tasks, but they exhibit significant limitations in spatial reasoning for non-textual visual elements. The study generates 15 binary 15x15 grids, rendered as text symbols and filled squares, to test the transcription ability of Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking. Results show that text-recognition pathways significantly outperform visual pathways in spatial reasoning, with F1 scores notably higher in the text-symbol condition. Each model exhibits distinct failure modes in the squares condition, indicating inadequate spatial localization capabilities for non-textual visual elements. The study reveals performance differences between text-recognition and visual pathways in VLMs, offering new perspectives for improving models' spatial reasoning capabilities.
Deep Analysis
Background
Vision-language models (VLMs) have made significant advances in visual understanding tasks, capable of describing images, interpreting charts, reasoning about diagrams, and answering questions about visual content. However, their spatial reasoning capabilities for non-textual visual elements remain inadequate. The study generates 15 binary 15x15 grids, rendered as text symbols and filled squares, to test the transcription ability of Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking.
Core Problem
The study reveals limitations in VLMs' spatial reasoning for non-textual visual elements, particularly in the filled-squares condition, where all models' accuracy and F1 scores significantly drop. The study aims to explore differences in spatial reasoning performance between text-recognition and visual pathways.
Innovation
The study is the first to use simple binary grid experiments to reveal performance differences between text-recognition and visual pathways in VLMs. Results show that text-recognition pathways significantly outperform visual pathways in spatial reasoning, offering new perspectives for improving models' spatial reasoning capabilities.
Methodology
- �� Generate 15 binary 15x15 grids, rendered as text symbols and filled squares.
- �� Test the transcription ability of Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking.
- �� Compare spatial reasoning performance between text-recognition and visual pathways.
Experiments
The experimental design includes generating 15 binary 15x15 grids, rendered as text symbols and filled squares. It tests the transcription ability of Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking, comparing spatial reasoning performance between text-recognition and visual pathways.
Results
Results show that text-recognition pathways significantly outperform visual pathways in spatial reasoning, with F1 scores notably higher in the text-symbol condition. Each model exhibits distinct failure modes in the squares condition, indicating inadequate spatial localization capabilities for non-textual visual elements.
Applications
The study's findings are significant for improving VLMs' spatial reasoning capabilities, particularly in applications requiring precise spatial localization, such as medical imaging, autonomous systems, and scientific visualization.
Limitations & Outlook
The study is limited to 15x15 binary grids, which may not fully reflect models' performance in complex scenarios. Experiments are conducted on three specific VLMs, which may not apply to other models.
Plain Language Accessible to non-experts
Imagine you're playing a simple crossword puzzle game where the board has some squares filled in black, while others are empty. Your task is to accurately identify which squares are black. Now, suppose you have an assistant who is very good at recognizing letters and numbers but struggles with simple black squares. This is the problem found in the study: vision-language models excel at recognizing text but struggle with simple visual elements. It's like your assistant quickly reacts to letters but gets confused when seeing black squares without letters.
ELI14 Explained like you're 14
Imagine you're playing a puzzle game where some squares are black and some are white. Your task is to find all the black squares. Now, suppose you have a super-smart robot helper that's great at recognizing letters and numbers but a bit clumsy with simple black squares. That's the problem found in the study: these super-smart models are great at recognizing text but a bit unreliable with simple visual elements. It's like your robot helper quickly reacts to letters but gets a bit confused when it sees black squares without letters.
Glossary
Vision-Language Model
A type of AI model that combines visual and language processing capabilities to understand and generate text related to visual content.
Used in the paper to test models' spatial reasoning under different visual conditions.
Spatial Reasoning
The ability to understand and infer the positions and relationships of objects in space.
Used in the study to evaluate models' performance under different visual conditions.
Text-Recognition Pathway
The pathway in a model used to recognize and process text elements in images, typically showing high accuracy.
Used to compare performance differences between text-recognition and visual pathways.
Visual Pathway
The pathway in a model used to process non-textual visual elements, typically weaker in spatial reasoning.
Used to compare performance differences between text-recognition and visual pathways.
F1 Score
A metric used to evaluate model performance, combining precision and recall.
Used to compare models' performance under different visual conditions.
Open Questions Unanswered questions from this research
- 1 How can the spatial reasoning capabilities of the visual pathway be improved for better performance on non-textual visual elements?
- 2 Can the performance gap between text-recognition and visual pathways be narrowed by improving training data?
Applications
Immediate Applications
Medical Imaging Analysis
Improve VLMs' spatial reasoning capabilities in medical imaging to enhance diagnostic accuracy.
Long-term Vision
Autonomous Systems
Enhance VLMs' spatial reasoning capabilities in complex visual scenarios to improve the safety and reliability of autonomous systems.
Abstract
We present a simple experiment that exposes a fundamental limitation in vision-language models (VLMs): the inability to accurately localize filled cells in binary grids when those cells lack textual identity. We generate fifteen 15x15 grids with varying density (10.7%-41.8% filled cells) and render each as two image types -- text symbols (. and #) and filled squares without gridlines -- then ask three frontier VLMs (Claude Opus, ChatGPT 5.2, and Gemini 3 Thinking) to transcribe them. In the text-symbol condition, Claude and ChatGPT achieve approximately 91% cell accuracy and 84% F1, while Gemini achieves 84% accuracy and 63% F1. In the filled-squares condition, all three models collapse to 60-73% accuracy and 29-39% F1. Critically, all conditions pass through the same visual encoder -- the text symbols are images, not tokenized text. The text-vs-squares F1 gap ranges from 34 to 54 points across models, demonstrating that VLMs behave as if they possess a high-fidelity text-recognition pathway for spatial reasoning that dramatically outperforms their native visual pathway. Each model exhibits a distinct failure mode in the squares condition -- systematic under-counting (Claude), massive over-counting (ChatGPT), and template hallucination (Gemini) -- but all share the same underlying deficit: severely degraded spatial localization for non-textual visual elements.