HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents
HyMobileAgent achieves efficient GUI interaction via data-environment co-scaling, 82.6% success on AndroidWorld.
Key Findings
Methodology
HyMobileAgent leverages the Hy3.0-VL-A3B model with a data-environment co-scaling framework. This includes a GUI perception flywheel, knowledge extraction pipeline, million-scale action data pipeline, and PhoneWorld mock environment, supporting long-horizon decision-making and dead-loop detection.
Key Results
- On the AndroidWorld benchmark, HyMobileAgent achieved an 82.6% success rate, surpassing larger models like Gemini 3.1.
- On the HyMobileWorld benchmark, it achieved a 42% success rate, comparable to Seed 2.0 Pro and far exceeding other same-scale models.
- Scored 89.3 on MMBench-GUI L2 and 96.2 on ScreenSpot V2, demonstrating superior performance.
Significance
This study highlights the importance of data-environment co-scaling in enhancing the efficiency and reliability of mobile GUI agents. It addresses bottlenecks in complex interface perception, data acquisition, and long-horizon decision-making, offering new insights for academia and industry.
Technical Contribution
HyMobileAgent introduces a novel approach by combining data and environment scaling, enhancing model generalization and execution efficiency. It integrates a GUI perception flywheel and PhoneWorld mock environment, setting a new technical path beyond existing methods.
Novelty
HyMobileAgent is the first to integrate data-environment scaling for mobile GUI agents, breaking the limitations of model scaling alone and offering a fresh research perspective.
Limitations
- The success rate on real devices still has room for improvement, especially in complex task scenarios.
- Adaptability to environmental changes needs further validation.
Future Work
Future research could explore more complex environment simulations and more efficient data scaling methods to further enhance model generalization and execution efficiency.
AI Executive Summary
As multimodal systems transition from content understanding to digital environment operation, mobile GUIs have become a critical testbed for digital intelligence. Existing methods face challenges in complex interface perception, data acquisition, and long-horizon decision-making. HyMobileAgent offers a novel solution through a data-environment co-scaling framework, leveraging the Hy3.0-VL-A3B model. Key technologies include a GUI perception flywheel, knowledge extraction pipeline, and PhoneWorld mock environment, supporting long-horizon decision-making and dead-loop detection. Experimental results show HyMobileAgent excels on AndroidWorld and HyMobileWorld benchmarks, achieving success rates of 82.6% and 42%, respectively. This research not only provides new directions for academia but also offers practical applications for industry. Despite its success, the model's performance on real devices still has room for improvement, suggesting future research could optimize data and environment scaling methods further.
Deep Analysis
Background
With the evolution of multimodal systems, mobile GUIs have become a crucial testbed for digital intelligence. Traditional methods face challenges in complex interface perception, data acquisition, and long-horizon decision-making. Recent advances in large language models and vision-language models have made progress but still have limitations.
Core Problem
Mobile GUI agents face challenges in complex interface perception, data acquisition, and long-horizon decision-making. These issues are interrelated, and relying solely on model scaling is insufficient to address them.
Innovation
HyMobileAgent offers a novel solution through a data-environment co-scaling framework, leveraging the Hy3.0-VL-A3B model. Its innovations include combining a GUI perception flywheel and PhoneWorld mock environment, enhancing model generalization and execution efficiency.
Methodology
- �� Utilizes the Hy3.0-VL-A3B model for perception and reasoning.
- �� GUI perception flywheel improves grounding capability through synthetic interface generation, rejection sampling, and icon-specific augmentation.
- �� Knowledge extraction pipeline transforms tutorial videos into structured interaction data.
- �� PhoneWorld mock environment provides a resettable training environment, supporting large-scale interaction trajectory collection.
Experiments
Experiments were conducted on AndroidWorld and HyMobileWorld benchmarks to evaluate the model's performance on real devices. Perception capabilities were tested using datasets like MMBench-GUI L2 and ScreenSpot V2.
Results
HyMobileAgent achieved an 82.6% success rate on the AndroidWorld benchmark and a 42% success rate on the HyMobileWorld benchmark. It scored 89.3 on MMBench-GUI L2 and 96.2 on ScreenSpot V2.
Applications
HyMobileAgent can be used as an intelligent assistant on mobile devices, supporting the automation of complex tasks. Its environment simulation capabilities can be used for training and evaluating other intelligent agents.
Limitations & Outlook
The model's performance on real devices still has room for improvement, especially in complex task scenarios. Adaptability to environmental changes needs further validation.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and HyMobileAgent is like a smart assistant that helps you identify ingredients, provides recipes, and hands you tools when needed. It not only understands your instructions but also adjusts plans based on changes in the environment, just like you would adjust a recipe based on available ingredients. By continuously learning new recipes and techniques, it can help you accomplish complex cooking tasks more effectively.
ELI14 Explained like you're 14
Imagine you're playing a complex mobile game, and HyMobileAgent is like a super assistant that helps you complete various tasks. It can recognize different icons in the game, just like you can recognize characters in the game. It can also adjust strategies based on game progress, just like you would change tactics based on enemy movements. By continuously learning new game skills, it can help you achieve higher scores in the game!
Glossary
Multimodal Model
A model capable of processing multiple data types, such as images and text.
Used for understanding and operating digital environments.
GUI Agent
An intelligent agent that performs tasks on a graphical user interface.
Used for task automation on mobile devices.
Data Scaling
A method to improve model performance by increasing data volume and diversity.
Used to enhance model generalization capabilities.
Environment Simulation
A method to create virtual environments for testing and training intelligent agents.
Used for training and evaluating HyMobileAgent.
Reinforcement Learning
A method of training agents through reward mechanisms.
Used to optimize the model's long-horizon decision-making capabilities.
Open Questions Unanswered questions from this research
- 1 How to improve HyMobileAgent's success rate on real devices, especially in complex task scenarios?
- 2 How to enhance the model's adaptability to environmental changes?
Applications
Immediate Applications
Intelligent Assistant
HyMobileAgent can be used as an intelligent assistant on mobile devices, helping users complete complex tasks.
Long-term Vision
Automated Workflows
Through environment simulation and data scaling, HyMobileAgent can be used to automate complex workflows, improving productivity.
Abstract
As large multimodal models move from understanding content to operating on digital environments, mobile GUI has emerged as a challenging and consequential testbed for digital embodied intelligence. Mobile agents operate under three coupled constraints: precise perception of complex interfaces, scalable acquisition of high-quality interaction data, and robust long-horizon decision making under compounding execution errors. This report presents HyMobileAgent, a mobile GUI agent built on Hy3.0-VL-A3B, a vision-native foundation model featuring native any-resolution input, an A3B-scale deployment budget, and a 32K context window to model extended interaction histories. Rather than relying solely on model scaling, we develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction. Our framework integrates a GUI perception flywheel combining mock-interface synthesis, rejection sampling, and icon-specific augmentation; a knowledge pipeline that transforms tutorial videos into structured interaction data; a million-scale action data pipeline deployed across more than 2000 sandbox and real-device instances with automated failure attribution; the PhoneWorld Mock App Factory, providing a resettable training environment with 34 mock applications and over 34000 tasks; and a structured Planning-and-Reflection mechanism with explicit dead-loop detection for reliable long-horizon execution. We also introduce a progressive training recipe consisting of mid-training, supervised fine-tuning, and reinforcement learning with task-specific reward designs.