SecAgent: Efficient Mobile GUI Agent with Semantic Context
SecAgent enhances mobile GUI agent efficiency with semantic context and a Chinese dataset, matching 7B-8B model performance.
Key Findings
Methodology
SecAgent employs a semantic context mechanism to convert historical screenshots and actions into concise natural language summaries, reducing computational costs. It is fine-tuned using supervised and reinforcement learning to enhance navigation performance.
Key Results
- SecAgent achieves 96.4% step accuracy on the CMGUI benchmark, significantly outperforming models of similar scale.
- On AndroidControl and GUIOdyssey benchmarks, SecAgent outperforms other 3B models and approaches 8B models.
- Ablation studies confirm the effectiveness of the semantic context mechanism in reducing computational overhead while maintaining information integrity.
Significance
SecAgent addresses the scarcity of multilingual data and inefficient history representation, advancing research on multilingual GUI agents with efficient history information representation and a Chinese dataset.
Technical Contribution
Introduces a novel history representation mechanism, the semantic context mechanism, significantly reducing computational costs and demonstrating effectiveness in Chinese mobile applications.
Novelty
First to introduce semantic context mechanism in mobile GUI agents, combined with a Chinese dataset, enhancing navigation performance in multilingual environments.
Limitations
- In complex interfaces, the semantic context may not capture all necessary information.
- Applicability to non-Chinese applications remains to be verified.
Future Work
Future work includes expanding to more languages and application scenarios, optimizing the semantic context mechanism for more complex interfaces.
AI Executive Summary
Mobile GUI agents show great potential in automating smartphone tasks, but existing methods face challenges of multilingual data scarcity and inefficient history representation. SecAgent enhances agent efficiency and performance by building a Chinese mobile GUI dataset and introducing a semantic context mechanism.
SecAgent excels on the CMGUI benchmark with a step accuracy of 96.4% and performs well on other benchmarks, approaching the performance of larger models. Its core lies in converting historical information into concise natural language summaries, reducing computational costs.
While SecAgent succeeds in the Chinese context, its applicability to other languages and more complex interfaces requires further study. Future work will focus on expanding application scenarios and optimizing algorithms to tackle broader challenges.
Deep Analysis
Background
In recent years, multimodal large language models have significantly advanced in mobile GUI agents. These agents, combining visual perception and natural language processing, can automate complex smartphone tasks. However, existing research primarily focuses on the English ecosystem, with limited attention to non-English environments.
Core Problem
The core problem for mobile GUI agents is the scarcity of multilingual data and inefficient history representation. Existing methods rely on platform-specific parsing, struggling to adapt to diverse UI designs, and complete historical information leads to prohibitive computational overhead.
Innovation
SecAgent's core innovation is the introduction of the semantic context mechanism, which converts historical screenshots and actions into natural language summaries, significantly reducing computational costs. Additionally, a human-verified Chinese mobile GUI dataset was constructed, filling the gap in multilingual data.
Methodology
- �� Construct a Chinese mobile GUI dataset with 18k samples and 121k navigation steps.
- �� Introduce a semantic context mechanism to dynamically maintain a natural language summary of historical information.
- �� Fine-tune the model using supervised and reinforcement learning to enhance navigation performance.
Experiments
Experiments were conducted on CMGUI, AndroidControl, and GUIOdyssey benchmarks to evaluate SecAgent's navigation performance. Step accuracy and task accuracy were used as evaluation metrics, and ablation studies were conducted to verify the effectiveness of the semantic context mechanism.
Results
SecAgent achieves 96.4% step accuracy on the CMGUI benchmark, significantly outperforming models of similar scale. On other benchmarks, it approaches 8B model performance, validating the effectiveness of the semantic context mechanism.
Applications
SecAgent can be used for automating tasks in multilingual mobile applications, particularly in the Chinese context. Its efficient history information representation mechanism allows it to perform well even on devices with limited computational resources.
Limitations & Outlook
SecAgent may face challenges in complex interfaces where information loss could occur, and its applicability to non-Chinese applications remains to be verified. Future work will focus on expanding application scenarios and optimizing algorithms.
Plain Language Accessible to non-experts
Imagine you're browsing a shopping app on your phone. Usually, you need to remember the products you've viewed and the links you've clicked, like keeping a shopping list in your mind. SecAgent acts like a smart assistant that automatically records this information and helps you find what you want faster without revisiting previous pages. It simplifies complex historical operations into easy-to-understand text summaries, making your shopping experience smoother.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a mobile game and need to remember clues and choices from each level. SecAgent is like a super memory helper that keeps track of all the important clues and choices, making it easier for you to find the right path in the game. It's like a smart notebook that turns complex info into simple text, helping you level up with ease!
Glossary
Semantic Context
Converts historical actions and screenshots into natural language summaries to reduce computational costs.
Used to dynamically maintain historical information for improved navigation efficiency.
Multimodal Large Language Model
Models combining visual and language processing capabilities for complex task automation.
Provides visual perception and language understanding for mobile GUI agents.
Reinforcement Learning
A machine learning method that optimizes model decision-making through reward mechanisms.
Used to fine-tune SecAgent for improved navigation performance.
CMGUI Dataset
A Chinese mobile GUI dataset with 18k samples and 121k navigation steps.
Used to train and evaluate SecAgent's multilingual navigation capabilities.
Step Accuracy
A metric measuring the correctness of each decision in navigation tasks.
Used to evaluate SecAgent's performance in benchmark tests.
Open Questions Unanswered questions from this research
- 1 How to maintain the completeness of semantic context in more complex interfaces?
- 2 How to extend SecAgent to other languages and application scenarios?
- 3 What is SecAgent's applicability in non-Chinese environments?
Applications
Immediate Applications
Automation in Chinese Mobile Apps
SecAgent can automate tasks in Chinese mobile apps, enhancing user experience and operational efficiency.
Long-term Vision
Multilingual Intelligent Assistant
In the future, SecAgent could become an intelligent assistant supporting multilingual environments, helping users navigate efficiently across apps in different languages.
Abstract
Mobile Graphical User Interface (GUI) agents powered by multimodal large language models have demonstrated promising capabilities in automating complex smartphone tasks. However, existing approaches face two critical limitations: the scarcity of high-quality multilingual datasets, particularly for non-English ecosystems, and inefficient history representation methods. To address these challenges, we present SecAgent, an efficient mobile GUI agent at 3B scale. We first construct a human-verified Chinese mobile GUI dataset with 18k grounding samples and 121k navigation steps across 44 applications, along with a Chinese navigation benchmark featuring multi-choice action annotations. Building upon this dataset, we propose a semantic context mechanism that distills history screenshots and actions into concise, natural language summaries, significantly reducing computational costs while preserving task-relevant information. Through supervised and reinforcement fine-tuning, SecAgent outperforms similar-scale baselines and achieves performance comparable to 7B-8B models on our and public navigation benchmarks. Our dataset is available at https://huggingface.co/datasets/alibabagroup/CMGUI.