VISTA: View-Consistent Self-Verified Training for GUI Grounding
VISTA method improves GUI grounding accuracy via multi-view comparison, significantly enhancing Qwen3-VL model performance.
Key Findings
Methodology
VISTA is a GRPO-based training framework that constructs comparison groups from multiple target-preserving views, comparing semantically equivalent but geometrically different inputs. It introduces self-verified cross-view anchors optimized with advantage-weighted loss.
Key Results
- On ScreenSpot-Pro, VISTA increases Qwen3-VL 4B/8B/30B-A3B accuracy from 55.5/52.7/53.7 to 63.4/65.8/67.0.
- Across five benchmarks, VISTA significantly improves grounding accuracy, especially on complex interfaces.
- VISTA combined with MVP method further enhances model robustness and accuracy.
Significance
VISTA addresses reward degeneracy in GUI grounding with multi-view comparison and self-verification, significantly improving accuracy on complex interfaces, impacting academia and industry.
Technical Contribution
VISTA overcomes limitations of traditional GRPO methods through multi-view comparison and self-verified anchors, offering new theoretical guarantees and engineering possibilities, significantly enhancing accuracy.
Novelty
VISTA is the first to introduce multi-view comparison and self-verified anchors in GUI grounding, addressing reward degeneracy with significant innovation compared to existing methods.
Limitations
- Coordinate generation may be unstable under complex views, affecting training effectiveness.
- Multi-view comparison increases computational costs.
- Self-verification mechanism may not activate in some scenarios.
Future Work
Future work could explore more efficient multi-view generation methods and further optimize self-verification mechanisms to enhance model robustness and accuracy.
AI Executive Summary
VISTA is an innovative training framework that significantly improves GUI grounding accuracy through multi-view comparison and self-verification mechanisms. Existing methods often face reward degeneracy issues on complex interfaces, leading to loss of learning signals.
VISTA addresses this by constructing comparison groups from multiple views, keeping target elements visible and comparing semantically equivalent but geometrically different inputs. The self-verified cross-view anchor further stabilizes coordinate generation.
Experimental results show that VISTA significantly improves grounding accuracy across multiple benchmarks, especially on complex interfaces. Combined with the MVP method, VISTA further enhances model robustness and accuracy, making a significant impact on academia and industry.
Deep Analysis
Background
GUI grounding enables autonomous agents to interact with digital interfaces, mapping screenshots and natural language instructions to click coordinates. Recent advances have been made using supervised fine-tuning and reinforcement learning, but reward degeneracy remains a challenge.
Core Problem
Existing methods face reward degeneracy on complex interfaces, leading to loss of learning signals and affecting grounding accuracy. Constructing effective comparison groups is key.
Innovation
VISTA addresses reward degeneracy with multi-view comparison and self-verification mechanisms. Multi-view comparison keeps target elements visible and compares semantically equivalent but geometrically different inputs. Self-verified anchors stabilize coordinate generation.
Methodology
- �� Construct comparison groups from multiple target-preserving views
- �� Introduce self-verified cross-view anchors optimized with advantage-weighted loss
- �� Combine with MVP method to enhance model robustness
Experiments
Experiments conducted on five benchmarks using Qwen3-VL models, comparing VISTA with existing methods. Focus on the impact of multi-view comparison and self-verification mechanisms.
Results
VISTA significantly improves grounding accuracy, especially on complex interfaces. Combined with MVP method, further enhances model robustness and accuracy.
Applications
VISTA can be applied in automated testing, interface design optimization, significantly improving interaction accuracy and efficiency.
Limitations & Outlook
Multi-view comparison increases computational costs. Self-verification mechanism may not activate in some scenarios, requiring further optimization.
Plain Language Accessible to non-experts
Imagine a kitchen where the chef needs to find a specific spice based on a recipe. Existing methods are like looking from one angle, possibly missing the correct spice. VISTA is like allowing the chef to view the kitchen from multiple angles, ensuring the correct spice is found. The self-verification mechanism is like the chef confirming the spice's correctness, ensuring no mistakes.
ELI14 Explained like you're 14
Imagine you're playing a game and need to click a button on the screen. Existing methods are like looking from one angle, possibly clicking the wrong button. VISTA is like letting you view the screen from multiple angles, ensuring you click the right button. The self-verification mechanism is like the game telling you you clicked correctly, ensuring no mistakes.
Glossary
Multi-view Comparison
Comparison through multiple views to enhance model robustness and accuracy.
Used in VISTA for constructing comparison groups.
Self-verified Anchor
A mechanism that stabilizes training by verifying model-generated results.
Used in VISTA to stabilize coordinate generation.
Reward Degeneracy
Loss of reward signals leading to decreased learning effectiveness.
A common issue in existing methods on complex interfaces.
MVP Method
An inference-time multi-view aggregation method that enhances model robustness.
Used in VISTA to improve model performance.
Qwen3-VL Model
A model used for GUI grounding with high accuracy.
Benchmark model used in VISTA.
Open Questions Unanswered questions from this research
- 1 How to further optimize multi-view generation methods to enhance model robustness?
- 2 Self-verification mechanism may not activate in some scenarios, how to address this issue?
Applications
Immediate Applications
Automated Testing
Improves interaction accuracy and efficiency, reducing manual testing costs.
Long-term Vision
Interface Design Optimization
Optimizes interface design by improving grounding accuracy, enhancing user experience.
Abstract
When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rollouts are sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage. We propose VISTA (View-Consistent Self-Verified Training), a GRPO-based training framework that constructs each comparison group from multiple target-preserving views of the same GUI instance.Each view is generated by a crop that keeps the target element visible and remaps its box exactly, so model rollouts are compared across semantically equivalent but geometrically different inputs. To stabilize short coordinate generation without turning reinforcement learning into unconditional imitation, VISTA further adds a self-verified cross-view anchor: an oracle answer optimized with an advantage-weighted loss, excluded from the group baseline and activated only when the model has produced a maximum-reward rollout. Across five GUI-grounding benchmarks and multiple Qwen backbones, VISTA consistently improves grounding accuracy.On ScreenSpot-Pro, it raises Qwen3-VL 4B/8B/30B-A3B from 55.5/52.7/53.7 to 63.4/65.8/67.0. Robustness analyses further show higher worst-view accuracy and lower prediction flip rates.