UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning
UI-Ins enhances GUI grounding with multi-perspective reasoning, achieving 87.3% on UI-I2E-Bench.
Key Findings
Methodology
The study introduces the 'Instruction-as-Reasoning' paradigm with a two-stage training framework: supervised fine-tuning on synthesized diverse instructions to instill multi-perspective reasoning, followed by reinforcement learning to optimize pathway selection and composition.
Key Results
- UI-Ins-32B achieved 87.3% accuracy on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2.
- UI-Ins-7B as an executor achieved a 74.1% success rate on AndroidWorld.
- The model effectively selects the most efficient reasoning pathway, significantly improving performance.
Significance
By introducing multi-perspective instruction reasoning, this research significantly enhances GUI grounding accuracy, addressing the overlooked impact of instruction diversity and quality on performance, providing new insights for intelligent GUI agent development.
Technical Contribution
The study introduces a novel 'Instruction-as-Reasoning' paradigm, combining supervised fine-tuning and reinforcement learning, overcoming limitations in existing methods' instruction handling, and offering new theoretical guarantees and engineering possibilities.
Novelty
This is the first to treat instructions as dynamic reasoning pathways rather than static inputs, significantly enhancing adaptability to diverse instructions.
Limitations
- In complex scenarios, the model may fail to select the optimal reasoning pathway, leading to performance drops.
- Instruction quality in datasets still requires improvement.
Future Work
Future work could explore more complex reasoning pathway selection mechanisms and validate in more real-world applications.
AI Executive Summary
In the field of GUI grounding, traditional methods often overlook the impact of instruction diversity and quality on performance. UI-Ins introduces the 'Instruction-as-Reasoning' paradigm, using multi-perspective instructions as dynamic reasoning pathways to significantly enhance grounding accuracy. The method employs a two-stage training framework, first conducting supervised fine-tuning on synthesized diverse instructions, then optimizing pathway selection through reinforcement learning. Experimental results show that UI-Ins achieves leading performance across multiple benchmarks, notably 87.3% accuracy on UI-I2E-Bench. This research not only holds significant academic value but also offers new directions for industrial GUI agent development. However, the model's performance in complex scenarios still requires optimization, and future research could explore more sophisticated reasoning mechanisms and real-world applications.
Deep Analysis
Background
GUI grounding is a crucial research direction in AI, aiming to map natural language instructions to actionable UI elements. Traditional methods often treat instructions as static inputs, overlooking the impact of instruction diversity and quality on grounding performance.
Core Problem
Existing methods perform poorly in handling diverse instructions, failing to effectively utilize different instruction perspectives for reasoning, leading to insufficient grounding accuracy.
Innovation
UI-Ins introduces the 'Instruction-as-Reasoning' paradigm, treating instructions as dynamic reasoning pathways, utilizing multi-perspective instructions for pathway selection, significantly enhancing adaptability to diverse instructions.
Methodology
- �� Supervised Fine-Tuning: Train on synthesized diverse instructions to enhance multi-perspective reasoning.
- �� Reinforcement Learning: Optimize reasoning pathway selection using GRPO to improve performance across scenarios.
Experiments
Experiments used multiple public datasets, including OS-Atlas and AMEX, employing UI-Ins-7B and UI-Ins-32B models to evaluate performance across different benchmarks.
Results
UI-Ins-32B achieved 87.3% accuracy on UI-I2E-Bench, significantly outperforming existing methods. The model also excelled in other benchmarks.
Applications
The method can be applied to intelligent GUI agent development, enhancing human-computer interaction efficiency and accuracy, especially in complex application scenarios.
Limitations & Outlook
The model may face challenges in selecting reasoning pathways in complex scenarios, requiring further optimization of reasoning mechanisms.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, where the recipe is like the instruction and the ingredients are the UI elements. Traditional methods are like glancing at the recipe title, while UI-Ins reads each step carefully, choosing the best cooking method. This way, even if the recipe is described in various ways, UI-Ins finds the best cooking path to make a delicious dish.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to find hidden treasure based on clues. Old methods are like reading just the first line of the clue, but UI-Ins is like a smart friend who analyzes each clue, finding the most effective hints to help you quickly find the treasure. Isn't that cool?
Glossary
GUI Grounding
The process of mapping natural language instructions to actionable elements in a graphical user interface.
Core task of the study.
Instruction-as-Reasoning
A paradigm treating instructions as dynamic reasoning pathways to enhance adaptability to diverse instructions.
New paradigm proposed in the study.
Supervised Fine-Tuning
Training on synthesized data to enhance specific model capabilities.
First stage of the training framework.
Reinforcement Learning
A learning method that optimizes model decisions through reward mechanisms.
Second stage of the training framework.
GRPO
A policy optimization method used in reinforcement learning.
Used to optimize reasoning pathway selection.
Open Questions Unanswered questions from this research
- 1 How to select the optimal reasoning pathway in complex scenarios? Current methods still need improvement in this area.
- 2 How to further improve instruction quality in datasets to reduce training noise?
Applications
Immediate Applications
Smart Assistant
Can be used to develop smarter GUI assistants, enhancing human-computer interaction efficiency.
Long-term Vision
Office Automation
In the future, it could transform office automation, enhancing productivity.
Abstract
GUI grounding, which maps natural-language instructions to actionable UI elements, is a core capability of GUI agents. Prior works largely treats instructions as a static proxy for user intent, overlooking the impact of instruction diversity and quality on grounding performance. Through a careful investigation of existing grounding datasets, we find a 23.3% flaw rate in their instructions and show that inference-time exploitation of instruction diversity yields up to a substantial 76% relative performance improvement. In this paper, we introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives and enabling the model to select the most effective pathway during reasoning. To achieve this, we propose a two-stage training framework: supervised fine-tuning (SFT) on synthesized, diverse instructions to instill multi-perspective reasoning, followed by reinforcement learning (RL) to optimize pathway selection and composition. Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks and exhibit emergent reasoning, selectively composing and synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model demonstrates strong agentic potential, achieving a 74.1% success rate on AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals additional insights such as how reasoning can be formulated to enhance rather than hinder grounding performance, and how our method mitigates policy collapse in the SFT+RL framework. All code and model checkpoints will be publicly released in https://github.com/alibaba/UI-Ins.