UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents
UI-KOBE enhances lightweight GUI agents with app-specific graph knowledge, boosting task success rate to 77.6%.
Key Findings
Methodology
UI-KOBE constructs an app-specific knowledge graph through autonomous exploration, where nodes represent UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph for guidance, identifying the current node and selecting appropriate actions. This reduces the reasoning burden on lightweight models, improving task execution reliability.
Key Results
- On AndroidWorld, using the Qwen3.5-4B model, UI-KOBE achieved a success rate of 70.7%, significantly higher than the 58.6% without graph guidance.
- On the A3 benchmark, UI-KOBE with the Qwen3.5-Plus model achieved an ESAR of 84.8 and an Overall SR of 78, outperforming most single-model agents.
- Graph guidance significantly enhances lightweight models' task execution capabilities in dynamic UI states.
Significance
By combining app-specific knowledge graphs with lightweight models, UI-KOBE significantly enhances the efficiency and reliability of mobile GUI task execution. This approach not only reduces computational costs but also offers better privacy solutions, suitable for scenarios requiring localized processing.
Technical Contribution
UI-KOBE integrates app knowledge graphs with lightweight GUI agents, providing a new task execution framework. Compared to existing methods, it improves task execution capability by reducing reasoning burden and enhancing decision guidance.
Novelty
UI-KOBE is the first to use app knowledge graphs for guiding lightweight GUI agents, overcoming the limitations of traditional end-to-end reasoning and offering a new task execution paradigm.
Limitations
- When graph guidance is unavailable, the system falls back to a simple planner, which may lead to decreased performance.
- The graph construction process requires time and cost, although it can be amortized over multiple task executions.
Future Work
Future work could explore automating the graph construction process, further improving graph coverage and accuracy, and studying how to extend this approach to more application scenarios.
AI Executive Summary
Recent advances in mobile GUI agents have shown great potential for automating tasks, but most systems rely on large vision-language models, limiting their direct deployment on devices. UI-KOBE overcomes this limitation by constructing app-specific knowledge graphs to guide lightweight GUI agents in task execution.
UI-KOBE autonomously explores applications to build knowledge graphs containing UI states and executable transitions. At runtime, lightweight agents use these graphs to identify the current state and select appropriate actions, reducing reasoning burden and improving task execution reliability.
Experimental results demonstrate that UI-KOBE significantly improves task success rates across multiple benchmarks, especially in dynamic UI states. This approach not only reduces computational costs but also offers better privacy solutions, providing new insights for automating tasks in mobile applications.
Deep Analysis
Background
With the proliferation of mobile devices, automating GUI task execution has become a significant research area. Traditional methods rely on large vision-language models, which, although effective, have high computational costs and privacy issues, limiting their direct deployment on devices. Lightweight models, while having lower computational costs, struggle with long-term reasoning and planning, making them unsuitable for complex tasks.
Core Problem
Lightweight GUI agents are unreliable when planning and executing tasks end-to-end from screenshots, especially in dynamic UI states. The core issue is the limited capacity of lightweight models, which struggle with complex task reasoning and planning.
Innovation
UI-KOBE constructs app-specific knowledge graphs to guide lightweight GUI agents in task execution. Its innovation lies in decoupling knowledge acquisition from task execution, reducing reasoning burden and improving task execution reliability.
Methodology
- �� Autonomously explore applications to build knowledge graphs, with nodes representing UI states and edges representing transitions.
- �� Use graph guidance at runtime to identify current nodes and select actions.
- �� Provide fallback mechanisms to ensure robustness when graph guidance is unavailable.
Experiments
Experiments were conducted on AndroidWorld and A3 benchmarks using Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-Plus models. Evaluation metrics included task success rate, ESAR, and Overall SR, showing significant performance improvements with UI-KOBE.
Results
On AndroidWorld, UI-KOBE with the Qwen3.5-4B model achieved a success rate of 70.7%, significantly higher than the 58.6% without graph guidance. On the A3 benchmark, with the Qwen3.5-Plus model, ESAR reached 84.8 and Overall SR reached 78.
Applications
UI-KOBE is suitable for mobile application tasks requiring localized processing, especially in privacy-sensitive and computational cost-sensitive scenarios. Its graph-guided approach can be reused across multiple task executions, improving efficiency.
Limitations & Outlook
Although UI-KOBE significantly improves task execution performance, when graph guidance is unavailable, the system falls back to a simple planner, which may lead to decreased performance. Additionally, the graph construction process requires time and cost.
Plain Language Accessible to non-experts
Imagine a factory where workers need to assemble products based on different tasks. Traditional methods are like having workers figure out how to assemble each time from scratch, while UI-KOBE is like providing them with a detailed flowchart, guiding them step-by-step. This way, workers can complete tasks faster and better because they don't need to rethink the entire process each time. For lightweight models, this approach greatly reduces their workload, allowing them to focus more on specific operations rather than the entire task planning.
ELI14 Explained like you're 14
Imagine you're playing a mobile game where you have to click a lot of buttons to pass each level. Traditional methods are like figuring out how to pass each level on your own every time, while UI-KOBE is like having a guide that tells you what to do at each step. This way, you can pass levels faster without having to start from scratch each time. This method is especially good for devices with small memory because it doesn't need a lot of computing power and can protect your privacy too!
Glossary
GUI Agent
A software that automates tasks on a graphical user interface.
In this paper, GUI agents are used to automate mobile application tasks.
Knowledge Graph
A structured data representation where nodes represent entities and edges represent relationships.
UI-KOBE uses knowledge graphs to represent app UI states and transitions.
Lightweight Model
A machine learning model with low computational resource requirements, suitable for resource-constrained devices.
UI-KOBE aims to enhance the task execution capabilities of lightweight models.
End-to-End Reasoning
A direct reasoning process from input to output without relying on intermediate steps or external guidance.
Traditional GUI agents rely on end-to-end reasoning to execute tasks.
ESAR
Essential-State Achievement Rate, a metric measuring the achievement rate of key states during task execution.
Used in the A3 benchmark to evaluate task execution progress.
Open Questions Unanswered questions from this research
- 1 How to automate the graph construction process to improve efficiency and accuracy remains an area for further research.
- 2 Technical challenges in extending UI-KOBE's approach to more application scenarios.
Applications
Immediate Applications
Mobile App Automation
UI-KOBE can be used to automate common tasks in mobile applications, such as settings adjustments and information retrieval.
Long-term Vision
Privacy-Protecting Smart Assistant
With localized processing and graph guidance, UI-KOBE could become a privacy-protecting smart assistant, widely used on personal devices.
Abstract
Recent advances in mobile GUI agents have shown strong potential for automating mobile tasks, but most effective systems still depend on large vision-language models for screenshot understanding and long-horizon planning. Small GUI agents that can be deployed directly on mobile devices are more attractive for practical use, offering lower inference cost and better protection of sensitive on-device information. However, due to limited model capacity, such lightweight agents remain unreliable when planning and executing GUI tasks end-to-end from screenshots alone. We propose Knowledge-Oriented Behavior Exploration (\textbf{UI-KOBE}), a framework that improves lightweight mobile GUI agents with reusable app-specific graph knowledge. UI-KOBE first autonomously explores a mobile application and constructs an app knowledge graph, where nodes represent distinct UI states and edges represent executable transitions. At runtime, a lightweight GUI agent uses the graph as external guidance: given a user task and the current screenshot, it identifies the current graph node and selects among self-loop actions, neighboring transitions, task completion, or fallback free actions associated with that node. By supporting runtime decisions with app-specific graph guidance, UI-KOBE reduces the burden of end-to-end GUI planning and helps lightweight models perform mobile GUI tasks more effectively, offering a practical step toward efficient, interpretable, and privacy-conscious on-device GUI agents.