OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis
OpenMobile synthesizes environment-aware instructions and trajectories with policy switching, achieving 51.7% success on AndroidWorld.
Key Findings
Methodology
OpenMobile employs two main components: a scalable environment exploration pipeline that builds a global environment memory from interaction data, and a policy-switching strategy during trajectory rollout that alternates between expert and learner models. Environment exploration uses random walks, perceptual hashing, and semantic retrieval to cluster screens and extract functionalities, forming a structured memory. Task synthesis combines short-term neighbor and long-term related functionalities via a vision-language model (e.g., Qwen-VL) to generate grounded, complex instructions. During trajectory collection, the policy-switching mechanism monitors deviation, triggering expert correction when necessary, thus capturing error-recovery data. The dataset includes 2800 instructions and 34000 steps across 20 Android apps, trained models outperform open-source baselines and approach industry standards.
Key Results
- Models fine-tuned on OpenMobile data achieved 51.7% and 64.7% success rates on AndroidWorld with Qwen2.5-VL and Qwen3-VL, respectively, surpassing existing open-data approaches (~30%) and nearing closed-source systems (~70%). In MobileWorld, performance improved from 9.4% to 17.4%. The policy switching strategy significantly enhanced error recovery, leading to more robust performance in complex tasks. The grounded, diverse instructions contributed to these gains, validated through ablation studies and overlap analysis.
Significance
This work addresses the critical bottleneck of closed training data in mobile agent development by providing an open-source, scalable data synthesis framework. It advances the understanding of how environment exploration and grounded instruction generation can improve task diversity and robustness. The approach democratizes access to high-quality training data, enabling broader research and practical deployment of autonomous mobile agents. It also bridges the gap between academic research and industry applications, fostering innovation in intelligent automation, user assistance, and cross-platform task execution.
Technical Contribution
The paper introduces a decoupled environment exploration and task synthesis pipeline, leveraging global environment memory and multi-level functionality retrieval to produce grounded instructions. It innovates with a policy-switching trajectory rollout that captures error correction signals, enhancing error recovery and robustness. The combination of perceptual hashing, semantic retrieval, and grounded instruction generation constitutes a novel framework that significantly improves data diversity and quality. Extensive experiments validate the effectiveness of these techniques across multiple benchmarks, setting new standards for open-source mobile agent training data.
Novelty
This is the first work to decouple environment exploration from task instruction synthesis in mobile agents, enabling the generation of diverse, grounded instructions beyond single-trajectory constraints. The introduction of a policy-switching mechanism for trajectory collection, which actively captures error-recovery signals, is a novel contribution that enhances robustness. These innovations collectively provide a scalable, transparent alternative to proprietary data pipelines, filling a critical gap in open-source mobile agent research.
Limitations
- Despite the improvements, the random exploration strategy may be inefficient in complex environments, limiting scalability. The policy switching relies on real-time deviation detection, which may not generalize well to highly dynamic scenarios. Additionally, the current models are trained in simulated environments, and their transferability to real-world devices requires further validation. Computational costs for large-scale data synthesis and model fine-tuning remain significant, posing challenges for broader adoption.
Future Work
Future research will integrate more intelligent exploration strategies, such as active learning and reinforcement learning, to improve efficiency. Enhancing environment adaptability and extending the framework to multi-platform scenarios will be prioritized. Incorporating continual learning mechanisms can help models adapt to evolving environments. Additionally, efforts will focus on deploying these models in real-world applications, such as assistive technologies and industrial automation, to realize their full potential.
AI Executive Summary
OpenMobile introduces a pioneering framework for building open-source mobile agents through environment exploration and task synthesis. By constructing a comprehensive global environment memory from interaction data, the system can generate diverse, grounded instructions that reflect real application functionalities. This approach addresses the core challenge of limited open data in mobile agent research, enabling the creation of high-quality, scalable datasets.
The framework’s second key innovation is a policy-switching trajectory rollout strategy. During data collection, the method alternates between expert and learner models, with a real-time monitor detecting deviations and triggering expert corrections. This process captures valuable error-recovery signals often missing in traditional imitation learning, significantly enhancing the robustness and error resilience of trained agents.
Empirical results demonstrate the effectiveness of OpenMobile: models fine-tuned on its data achieve 51.7% and 64.7% success rates on AndroidWorld, outperforming previous open-data baselines (~30%) and approaching industry-leading closed-source systems (~70%). The data synthesis process produces 2800 instructions and 34000 steps across 20 Android applications, covering complex, multi-step tasks with high groundedness and diversity.
These advancements have profound implications for both academia and industry. By providing a transparent, scalable pipeline for high-quality data generation, OpenMobile democratizes access to powerful training resources, fostering innovation in autonomous mobile systems. Its design principles can be extended to other domains requiring grounded instruction and robust error recovery, paving the way for more reliable, adaptable AI agents.
Despite these successes, challenges remain. The reliance on random exploration and real-time deviation detection may limit efficiency and generalizability. Future work will focus on integrating more sophisticated exploration algorithms, continual learning, and real-world deployment. Overall, OpenMobile marks a significant step toward autonomous, open, and scalable mobile AI systems.
Deep Analysis
Background
移动代理技术经历了从结构化界面理解到端到端视觉语言模型的快速演进。早期工作如Accessibility树的利用（Deng et al., 2023）逐步转向基于深度学习的GUI理解（Gou et al., 2024），行业内的代表性系统包括Operator（OpenAI, 2025）和UI-TARS（Qin et al., 2025），在Android环境中取得了接近70%的成功率。这些系统依赖大量封闭训练数据，缺乏公开的合成策略，限制了学术界的创新空间。近年来，开源社区如AndroidControl（Li et al., 2024）和AMEX（Chai et al., 2025）提供了基础数据，但存在噪声大、功能覆盖有限的问题，模型性能停滞在30%左右。
Core Problem
行业领先的移动代理系统普遍依赖封闭数据，缺乏公开的任务指令和轨迹生成机制，导致研究的可复现性和创新性受限。公开数据集虽提供基础，但存在标注噪声和功能有限的问题，阻碍模型性能的提升。如何在保证指令 grounded 和多样性的同时，增强模型的错误恢复能力，成为核心难题。传统方法多依赖单一轨迹，难以生成多样化、 grounded 的指令，且缺乏对错误修正的系统性支持。
Innovation
OpenMobile提出了环境探索与任务合成的解耦策略，利用全局环境记忆组织应用功能信息，结合短期邻近屏幕和长期语义相关功能，生成 grounded、多样的任务指令。引入基于策略切换的轨迹回放机制，通过实时偏差检测触发专家模型修正，丰富错误修正数据，提升模型鲁棒性。这些创新突破了传统单轨迹、封闭数据的局限，为开源社区提供了高效、可扩展的合成方案，推动了移动代理的研究进步。
Methodology
- �� 环境探索：采用随机游走收集应用状态转移数据，利用感知哈希聚类屏幕状态，提取功能集，构建全局记忆库。• 任务合成：结合短期邻近屏幕信息和长期语义相关功能，通过视觉语言模型（如Qwen-VL系列）在丰富上下文中生成 grounded 指令。• 轨迹回放：采用策略切换机制，结合专家模型和学习模型，实时监控偏差，触发专家干预，增强错误修正。• 数据合成：在20个Android应用中生成2800条指令、34000动作，确保多样性和 grounded 性。• 训练与评估：在AndroidWorld、AndroidLab、MobileWorld上验证模型性能，采用成功率和任务完成度作为指标。
Experiments
采用真实Android模拟环境，训练Qwen2.5-VL-7B和Qwen3-VL-8B模型，比较不同轨迹回放策略的效果。基准包括AndroidWorld、AndroidLab和MobileWorld，评估指标为Pass@1和Pass@3。通过 ablation 实验验证环境记忆和策略切换的贡献，分析合成指令的 grounded 性和多样性。模型参数设定为学习率1e-5，训练3轮，采用RL微调验证鲁棒性。
Results
模型在AndroidWorld达成51.7%和64.7%的成功率，显著优于开源数据集（约30%），接近闭源系统（70%）。在MobileWorld中，性能由9.4%提升至17.4%。引入策略切换后，错误修正能力增强，模型在复杂任务中表现更稳健。指令多样性和 grounded 性显著提升，验证了全局环境记忆和策略切换的有效性。
Plain Language Accessible to non-experts
想象你在厨房做饭。传统方法就像只记住一道菜的做法，遇到问题只能重做。OpenMobile像是你先把所有食材都整理好，记住每个食材的用途，然后根据需要灵活搭配，遇到难题还能请教厨师帮忙修正。它通过不断探索厨房的每个角落，建立一个完整的食材和工具的地图，之后用这个地图生成复杂的菜谱。做菜时，系统会监控你的操作，一旦偏离正确步骤，就请厨师帮忙纠正，确保菜肴最终成功。这种方法让厨房操作变得更智能、更可靠，也能做出更多不同的菜肴。
ELI14 Explained like you're 14
想象你在学校学做手工艺品。以前老师只教你一条固定的步骤，你照着做遇到问题就得自己猜。现在，老师先带你探索整个工作室，告诉你所有工具和材料都能做什么，然后你可以用这些信息自己设计出复杂的作品。过程中，如果你偏离了正确的步骤，老师会及时帮你纠正，确保你能完成作品。这就像是你有了一个详细的地图和一个聪明的老师，既能学会多样的技巧，又能在遇到困难时得到帮助。这样，做手工就变得更有趣，也更容易成功。
Glossary
Vision-Language Model (视觉语言模型)
结合视觉信息和文本理解的模型，用于理解UI界面和生成指令。In this paper, it用于环境理解和指令生成。
用于构建环境记忆和生成 grounded 指令。
Policy Switching (策略切换)
在轨迹回放中交替使用专家模型和学习模型，捕获错误修正信息。In this work, it增强模型的错误恢复能力。
提升轨迹的多样性和鲁棒性。
Global Environment Memory (全局环境记忆)
通过探索收集的应用状态和功能信息的结构化存储，用于指令生成。
支撑 grounded、多样化任务指令的合成。
Grounded Instruction ( grounded 指令)
基于环境状态和功能的具体任务描述，确保指令可执行。
提升指令的 grounded 性和任务成功率。
Perceptual Hashing (感知哈希)
一种图像相似性检测方法，用于屏幕状态聚类。
组织和组织环境状态，构建全局记忆库。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端复杂或长时间任务中的错误恢复能力，仍需探索更智能的策略和模型结构。
- 2 环境探索策略仍以随机游走为主，未来需结合主动探索和强化学习以提高效率。
- 3 模型对环境变化的适应性和持续学习能力有待增强，以应对真实世界的不断变化。
Applications
Immediate Applications
智能手机自动操作
基于OpenMobile的模型可以实现自动化操作，如自动设置、信息采集，减少用户手动干预，提升效率。
企业自动化测试
利用合成任务指令和轨迹，自动测试移动应用的功能，节省人力和时间，提升测试覆盖率。
Long-term Vision
自主智能助手
未来可发展为具备自主学习和适应能力的智能助手，能在多环境、多任务中自主完成复杂操作，推动智能家居和工业自动化。
Abstract
Mobile agents powered by vision-language models have demonstrated impressive capabilities in automating mobile tasks, with recent leading models achieving a marked performance leap, e.g., nearly 70% success on AndroidWorld. However, these systems keep their training data closed and remain opaque about their task and trajectory synthesis recipes. We present OpenMobile, an open-source framework that synthesizes high-quality task instructions and agent trajectories, with two key components: (1) The first is a scalable task synthesis pipeline that constructs a global environment memory from exploration, then leverages it to generate diverse and grounded instructions. and (2) a policy-switching strategy for trajectory rollout. By alternating between learner and expert models, it captures essential error-recovery data often missing in standard imitation learning. Agents trained on our data achieve competitive results across three dynamic mobile agent benchmarks: notably, our fine-tuned Qwen2.5-VL and Qwen3-VL reach 51.7% and 64.7% on AndroidWorld, far surpassing existing open-data approaches. Furthermore, we conduct transparent analyses on the overlap between our synthetic instructions and benchmark test sets, and verify that performance gains stem from broad functionality coverage rather than benchmark overfitting. We release data and code at https://njucckevin.github.io/openmobile/ to bridge the data gap and facilitate broader mobile agent research.