GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
GUI-Primitives benchmark reveals limitations in spatial reasoning of vision-language models, with max accuracy of 32%.
Key Findings
Methodology
The study introduces the GUI-Primitives benchmark, consisting of 994 contrastive instruction pairs covering seven spatial relations. Each pair keeps the screenshot and anchor fixed, only changing the relation expression to test models' ability to bind relational language in GUIs. Five annotators validated a 196-item subset to ensure data accuracy.
Key Results
- 19 vision-language models achieved a maximum strict point-in-box accuracy of 32%.
- Predictions fell outside both candidate regions in 60-92% of items.
- Marking the two designated candidates increased selection accuracy by 35-57 percentage points.
Significance
The study highlights the limitations of current vision-language models in GUI spatial reasoning, particularly in understanding relations. This provides a clear direction for future model improvements and sets a new evaluation standard for academia and industry.
Technical Contribution
The study provides a new benchmark tool that precisely diagnoses specific failure points in vision-language models' GUI spatial reasoning, particularly in candidate localization and relation understanding. Through contrastive analysis, it reveals performance differences across different spatial relations.
Novelty
This is the first systematic evaluation of vision-language models' spatial reasoning in GUIs using contrastive instruction pairs. Unlike previous benchmarks, GUI-Primitives can precisely isolate failure points in relational language binding.
Limitations
- Models' selection accuracy for containment and occlusion relations did not significantly exceed 0.50, reflecting deficiencies in relation understanding.
- Most models' predictions fell outside both candidate regions, indicating difficulties in candidate localization.
Future Work
Future research can explore enhancing models' spatial reasoning capabilities in complex GUI environments, particularly in improving relation understanding and candidate localization.
AI Executive Summary
In modern computing environments, vision-language models are used to parse natural language instructions in graphical user interfaces (GUIs). However, existing benchmarks fail to effectively isolate models' failure points in relational language binding. To address this, researchers introduced the GUI-Primitives benchmark, consisting of 994 contrastive instruction pairs covering seven spatial relations. By keeping the screenshot and anchor fixed and only changing the relation expression, researchers can test models' ability to bind relational language in GUIs. Experimental results show that 19 vision-language models achieved a maximum strict point-in-box accuracy of 32%. Most predictions fell outside both candidate regions, indicating difficulties in candidate localization. Marking the two designated candidates increased selection accuracy by 35-57 percentage points. This study highlights the limitations of current models in GUI spatial reasoning, providing a clear direction for future improvements.
Deep Analysis
Background
Vision-language models have made significant progress in natural language processing and computer vision in recent years. However, in graphical user interfaces (GUIs), models need to parse complex spatial relations, posing new challenges. Existing benchmarks like ScreenSpot and ScreenSpot-Pro measure models' accuracy in clicking within target boxes but fail to isolate specific failure points in spatial relation understanding.
Core Problem
Current vision-language models have limited spatial reasoning capabilities in GUIs, particularly in binding relational language to the correct elements. This makes it difficult for models to accurately locate target elements in complex GUI environments, affecting their practical application.
Innovation
The GUI-Primitives benchmark systematically evaluates models' spatial reasoning capabilities in GUIs using contrastive instruction pairs. Each pair keeps the screenshot and anchor fixed, only changing the relation expression to test models' ability to bind relational language. This method can precisely isolate failure points in relation understanding.
Methodology
- �� Introduce 994 contrastive instruction pairs covering seven spatial relations.
- �� Validate a 196-item subset with five annotators to ensure data accuracy.
- �� Use strict point-in-box accuracy to evaluate model performance.
- �� Increase selection accuracy by marking candidates.
Experiments
The experiments involved 19 vision-language models, including open-source and proprietary systems. The models' performance on the GUI-Primitives benchmark was evaluated using strict point-in-box accuracy and candidate marking to improve selection accuracy.
Results
Experimental results show that models achieved a maximum strict point-in-box accuracy of 32%. Most predictions fell outside both candidate regions, indicating difficulties in candidate localization. Marking the two designated candidates increased selection accuracy by 35-57 percentage points.
Applications
The GUI-Primitives benchmark can be used to evaluate and improve vision-language models' spatial reasoning capabilities in complex GUI environments, aiding in the development of smarter computer-use agents.
Limitations & Outlook
Models' selection accuracy for containment and occlusion relations did not significantly exceed 0.50, reflecting deficiencies in relation understanding. Most models' predictions fell outside both candidate regions, indicating difficulties in candidate localization.
Plain Language Accessible to non-experts
Imagine you're in a huge library looking for a book. You have a task: find the third book to the left on a specific shelf. This task is like how vision-language models find elements in a GUI. The model needs to understand the instruction 'third book to the left' and find the right book in a complex shelf layout. The GUI-Primitives benchmark is like a test to see if the model can find the book accurately. Results show that most models struggle with this task, like getting lost in the library.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a game where you need to find a hidden treasure on the screen. The game gives you hints like 'next to the tree on the left.' This is like how vision-language models find things in a GUI. Scientists used something called the GUI-Primitives test to see if these models can find the 'treasure.' Turns out, most models can't find the right place, just like getting lost in the game. Scientists hope this test will help models get smarter, like you finding all the treasures in the game!
Glossary
Vision-Language Model
A model that combines visual and language information to perform tasks, commonly used in image captioning and question answering.
In this paper, used to parse natural language instructions in GUIs.
GUI (Graphical User Interface)
An interface that allows users to interact with computers through graphical elements.
In this paper, models need to locate target elements in a GUI.
Point-in-box Accuracy
The accuracy of whether a predicted point falls within the target box.
Used to evaluate model performance on the GUI-Primitives benchmark.
Contrastive Instruction Pair
Instruction pairs that keep the screenshot and anchor fixed, only changing the relation expression.
Used to test models' ability to bind relational language.
Candidate Marking
A method of marking two designated candidates to improve selection accuracy.
Used in experiments to improve model selection accuracy.
Open Questions Unanswered questions from this research
- 1 How to enhance models' spatial reasoning capabilities in complex GUI environments, particularly in relation understanding and candidate localization.
- 2 Models' selection accuracy for containment and occlusion relations did not significantly exceed 0.50, reflecting deficiencies in relation understanding.
Applications
Immediate Applications
GUI Evaluation
Use the GUI-Primitives benchmark to evaluate current vision-language models' spatial reasoning capabilities, aiding in the development of smarter computer-use agents.
Long-term Vision
Intelligent Interface Interaction
By improving models' spatial reasoning capabilities, achieve more natural and intelligent human-computer interaction interfaces, enhancing user experience.
Abstract
Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset ($κ= 0.94$ well-formedness; $κ= 0.79$ target selection). Nineteen vision-language models reach at most $32\%$ strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on $60-92\%$ of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman $ρ= +0.74$), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.