Towards Scalable Lightweight GUI Agents via Multi-role Orchestration
LAMO framework enhances lightweight GUI agents' task scalability via multi-role orchestration, achieving 52.6% accuracy on ScreenSpot-pro.
Key Findings
Methodology
The LAMO framework enhances task scalability of lightweight MLLMs through role-oriented data synthesis and a two-stage training strategy. Stage one uses Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, while stage two employs reinforcement learning for role-oriented cooperative exploration.
Key Results
- LAMO-3B achieved 52.6% screen grounding accuracy on ScreenSpot-pro, significantly outperforming other lightweight models.
- On AndroidControl-High, LAMO-3B achieved a success rate of 77.1%, demonstrating strong task execution capabilities.
- In the MiniWob++ online environment, LAMO-3B performed excellently, validating its potential in multi-role systems.
Significance
This research opens new possibilities for lightweight GUI agents on resource-constrained devices, addressing the high deployment costs of large-scale models. By enabling multi-role orchestration, the LAMO framework enhances task scalability, allowing lightweight models to adapt to complex real-world scenarios.
Technical Contribution
The LAMO framework breaks the task scalability bottleneck of lightweight models through role-oriented data synthesis and a two-stage training strategy, offering new engineering possibilities, especially in multi-role systems.
Novelty
LAMO is the first framework to apply multi-role orchestration to lightweight GUI agents, significantly enhancing task scalability through role-oriented data synthesis and reinforcement learning.
Limitations
- In complex GUI layouts, LAMO-3B's screen grounding accuracy still has room for improvement.
- LAMO-3B's performance may be limited on extremely resource-constrained devices.
Future Work
Future research could explore the adaptability of the LAMO framework in more real-world applications and optimize its performance in extremely resource-constrained environments.
AI Executive Summary
Lightweight GUI agents face challenges on resource-constrained devices, with existing methods struggling to deploy effectively in real-world scenarios. The LAMO framework enhances lightweight MLLMs with GUI-specific knowledge and task scalability through multi-role orchestration, enabling participation in realistic GUI workflows. Its core technologies include role-oriented data synthesis and a two-stage training strategy, significantly improving task scalability. In experiments, LAMO-3B performed excellently across multiple benchmarks, particularly in environments like ScreenSpot-pro and AndroidControl. This research provides new insights for applying lightweight models in complex scenarios, though performance in extremely resource-constrained environments still requires optimization.
Deep Analysis
Background
In recent years, multimodal large models have made significant progress in GUI automation, but their high deployment costs limit application on resource-constrained devices. Lightweight GUI agents are gaining attention for their cost-effectiveness, but their task scalability and adaptability need improvement.
Core Problem
Lightweight GUI agents face task scalability issues in complex scenarios, struggling to adapt to multi-role system demands. Existing methods have bottlenecks in combining high-level reasoning and low-level execution.
Innovation
The LAMO framework enhances task scalability of lightweight models through role-oriented data synthesis and a two-stage training strategy. Its innovation lies in solving adaptability issues in complex scenarios through multi-role orchestration.
Methodology
- �� Role-oriented data synthesis: Decomposes GUI automation into five core capabilities. • Perplexity-Weighted Cross-Entropy optimization: Enhances visual perception. • Reinforcement learning: Optimizes role-oriented tasks through multi-task cooperative exploration.
Experiments
Experiments used benchmarks like ScreenSpot and AndroidControl to evaluate LAMO-3B's performance in various environments. The design included multi-role orchestration and online environment tests.
Results
LAMO-3B achieved 52.6% accuracy on ScreenSpot-pro and a 77.1% success rate on AndroidControl-High, validating its potential in multi-role systems.
Applications
The LAMO framework can be applied in scenarios requiring efficient GUI automation, such as mobile app management and desktop software operations, with broad industry impact.
Limitations & Outlook
LAMO-3B's performance in complex GUI layouts needs improvement, and future research could optimize its performance in extremely resource-constrained environments.
Plain Language Accessible to non-experts
Imagine shopping in a large supermarket with many different sections, each with specific functions. The LAMO framework acts like a smart shopping assistant that quickly identifies the function of each section and helps you efficiently complete your shopping tasks. Through role-oriented data synthesis and a two-stage training strategy, LAMO is like an experienced guide, providing the best shopping advice in different scenarios.
ELI14 Explained like you're 14
Hey kiddo! Imagine you're playing a super complex game with lots of different levels, each with different tasks. LAMO is like a super cool game assistant that helps you quickly identify each level's tasks and gives you the best strategies to win. With role-oriented data synthesis and a two-stage training strategy, LAMO is like an experienced game guide, offering the best strategies for different game scenarios.
Glossary
Multimodal Large Language Models
Models that combine multiple data modalities (e.g., text, images) to handle complex multimodal tasks.
Used in the LAMO framework to enhance GUI agents' task scalability.
Role-oriented Data Synthesis
Synthesizing data specific to task roles to improve model performance in specific tasks.
Used in the data preparation phase of the LAMO framework.
Perplexity-Weighted Cross-Entropy
An optimization method that weights perplexity to enhance visual perception.
Used in the first training stage of the LAMO framework.
Reinforcement Learning
A machine learning method that learns optimal strategies through interaction with the environment.
Used in the second training stage of the LAMO framework.
Multi-role Orchestration
Coordinating multiple roles to enhance system task scalability and adaptability.
A core innovation of the LAMO framework.
Open Questions Unanswered questions from this research
- 1 How can the LAMO framework's performance be optimized in extremely resource-constrained environments?
- 2 What is the adaptability of the LAMO framework in more real-world application scenarios?
Applications
Immediate Applications
Mobile App Management
Using the LAMO framework, users can more efficiently manage and operate apps on mobile devices, improving productivity.
Long-term Vision
Smart Home Management
The LAMO framework could be used for automated management of smart home devices, achieving a smarter home living experience.
Abstract
Autonomous Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) enable digital automation on end-user devices. While scaling both parameters and data has yielded substantial gains, advanced methods still suffer from prohibitive deployment costs on resource-constrained devices. When facing complex in-the-wild scenarios, lightweight GUI agents are bottlenecked by limited capacity and poor task scalability under end-to-end episodic learning, impeding adaptation to multi-agent systems (MAS), while training multiple skill-specific experts remains costly. Can we strike an effective trade-off in this cost-scalability dilemma, enabling lightweight MLLMs to participate in realistic GUI workflows? To address these challenges, we propose the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability, allowing multi-role orchestration to expand its capability boundary for GUI automation. LAMO combines role-oriented data synthesis with a two-stage training recipe: (i) supervised fine-tuning with Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, and (ii) reinforcement learning for role-oriented cooperative exploration. With LAMO, we develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration. When paired with advanced planners as a plug-and-play policy executor, LAMO-3B can continuously benefit from planner advances, enabling a higher performance ceiling. Extensive static and online evaluations validate the effectiveness of our design.