UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
UI-Voyager employs RFT and GRSD, achieving 81% success on AndroidWorld with a 4B model, surpassing human performance.
Key Findings
Methodology
UI-Voyager introduces a two-stage self-evolving framework: first, Rejection Fine-Tuning (RFT) iteratively filters and refines data and models through autonomous trajectory generation and rejection sampling; second, Group Relative Self-Distillation (GRSD) identifies critical fork points via SSIM-based matching among trajectory groups, extracting dense step-level supervision from successful trajectories to correct failures. The approach leverages multi-round filtering, fork point detection, and self-distillation, forming a closed-loop optimization. The backbone model is Qwen3-VL-4B-Instruct, and the method effectively addresses the credit assignment problem in long-horizon GUI tasks.
Key Results
- The 4B UI-Voyager achieves 81.0% Pass@1 success rate on AndroidWorld, outperforming baselines such as UI-Tars-7B (49.1%) and Qwen3-VL-8B (47.6%), and exceeds human-level performance (80%). Ablation studies confirm GRSD’s role in identifying fork points and improving stability, with success rates increasing from 37% to 81% over iterations.
- Multi-round rejection sampling and self-distillation significantly improve data efficiency and learning speed, demonstrating the model’s strong self-improvement capability.
- The method effectively mitigates credit sparse issues, enabling a compact 4B model to excel in complex, long-horizon tasks, setting new benchmarks for mobile GUI automation.
Significance
This work advances the field by overcoming the inefficiency of learning from failed trajectories and ambiguous credit assignment in long-horizon GUI tasks. It provides a scalable, automated approach to mobile GUI automation without manual annotation, facilitating deployment in real-world scenarios like mobile app testing, personal assistants, and accessibility tools. The self-evolving paradigm paves the way for autonomous agents capable of continuous improvement, reducing reliance on human supervision and expanding the scope of intelligent automation in dynamic environments.
Technical Contribution
The core innovations include the rejection-based data filtering (RFT) mechanism, enabling autonomous data-model co-evolution, and the group relative self-distillation (GRSD), which detects critical fork points and performs step-level supervision. These techniques transform sparse reward signals into dense, targeted feedback, effectively addressing credit assignment in long sequences. The framework combines multi-round trajectory generation, rule-based filtering, SSIM-based fork detection, and self-distillation, offering a novel paradigm for scalable, self-improving GUI agents that outperform traditional RL-based methods.
Novelty
This is the first work to incorporate group-based self-distillation with fork point detection for mobile GUI tasks, transforming sparse, trajectory-level rewards into dense, step-level supervision. Unlike prior methods relying solely on reward signals, this approach leverages successful peer trajectories within groups to self-correct, enabling efficient learning in complex, long-horizon environments. The integration of multi-round rejection sampling with self-distillation represents a significant methodological advance, setting new standards for autonomous GUI agents.
Limitations
- The fork point detection relies on SSIM similarity, which may be less accurate under extreme visual variations or complex UI states, potentially leading to incorrect supervision.
- Training involves extensive trajectory generation and filtering, incurring high computational costs, which could limit scalability.
- The current framework assumes stable visual and interaction environments; in highly dynamic or unseen scenarios, performance may degrade, requiring further robustness enhancements.
Future Work
Future directions include integrating multimodal signals (e.g., audio, text) for more robust fork detection, exploring unsupervised or semi-supervised methods to reduce computational costs, and extending the framework to multi-platform environments. Additionally, combining reinforcement learning with self-distillation could further enhance adaptability and generalization, pushing autonomous GUI agents towards more human-like intelligence.
AI Executive Summary
In recent years, the automation of mobile GUI tasks has become a critical area of research, driven by the proliferation of smartphones and the need for intelligent, autonomous interfaces. Traditional methods, based on rule-based systems or supervised learning, struggle with long-horizon tasks due to sparse rewards and the difficulty of learning from failures. Existing reinforcement learning approaches often face challenges in credit assignment, where the model cannot determine which specific actions led to success or failure, especially in complex, multi-step interactions.
To address these issues, this paper introduces UI-Voyager, a novel framework that combines Rejection Fine-Tuning (RFT) and Group Relative Self-Distillation (GRSD). RFT enables the model to autonomously generate and filter trajectories, continuously refining both data and model capabilities without manual annotations. This creates a self-evolving loop where the agent learns from its own experiences, improving efficiency and robustness. The second component, GRSD, identifies critical fork points within trajectory groups by matching shared states using SSIM, then distills successful actions from peer trajectories into failed ones, providing dense, step-level supervision. This approach effectively transforms sparse, trajectory-level rewards into precise, actionable feedback.
Experimental results on the AndroidWorld benchmark demonstrate the effectiveness of the method. The 4B parameter model achieves an 81.0% Pass@1 success rate, surpassing all baselines, including larger models and human performance (80%). Ablation studies confirm the importance of GRSD in identifying fork points and correcting errors, significantly boosting success rates. The combined approach not only improves data efficiency but also enhances the model’s ability to learn complex, long-horizon tasks autonomously.
This work marks a significant step forward in mobile GUI automation, offering a scalable, self-improving solution that reduces reliance on manual data annotation. Its implications extend to automated testing, personal assistants, and accessibility tools, where autonomous, adaptable agents can greatly enhance user experience. Future research will focus on integrating multimodal signals, reducing computational costs, and expanding to multi-platform scenarios, aiming to realize truly intelligent, autonomous GUI agents capable of continuous self-improvement in diverse environments.
Deep Dive
Glossary
Rejection Fine-Tuning (RFT)
一种自动筛选高质量轨迹并持续优化模型的机制，结合轨迹生成和筛选，提升学习效率。/ A mechanism that autonomously filters high-quality trajectories and iteratively refines the model, combining trajectory generation and rejection sampling.
Group Relative Self-Distillation (GRSD)
通过识别轨迹组中的叉点，将成功轨迹的动作迁移到失败轨迹，实现密集逐步监督。/ A method that detects fork points within trajectory groups and distills successful actions into failed trajectories for dense step-level supervision.
SSIM (Structural Similarity Index)
一种衡量两幅图像相似度的指标，用于匹配UI状态。/ A metric for measuring similarity between two images, used here to match UI states.
长序列信用分配 (Credit Assignment in Long Sequences)
在长序列任务中，模型难以识别哪些步骤导致失败，信用信号稀疏的问题。/ The challenge of determining which steps in a long sequence caused failure, due to sparse reward signals.
自我演化 (Self-Evolution)
模型通过自主生成、筛选和修正数据，不断提升性能的过程。/ The process where a model autonomously generates, filters, and corrects data to continuously improve.
Open Questions Unanswered questions from this research
- 1 如何在极端视觉变化或复杂交互环境中确保叉点检测的准确性仍是未解难题，未来需结合多模态信息提升鲁棒性。
- 2 模型在未覆盖新任务或环境时的泛化能力有限，需探索迁移学习和多任务训练策略以增强适应性。
Applications
Immediate Applications
移动端自动操作
可应用于自动化测试、个性化助手，提升操作效率，减少人工干预，适合企业和个人用户。
自动化测试工具
利用模型自主执行复杂测试流程，提升软件质量和发布速度，减少人力成本。
Long-term Vision
智能人机交互
未来实现更自然、智能的交互界面，模型能自主学习新任务，推动智能助手和无障碍技术发展。
Abstract
Autonomous mobile GUI agents have attracted increasing attention along with the advancement of Multimodal Large Language Models (MLLMs). However, existing methods still suffer from inefficient learning from failed trajectories and ambiguous credit assignment under sparse rewards for long-horizon GUI tasks. To that end, we propose UI-Voyager, a novel two-stage self-evolving mobile GUI agent. In the first stage, we employ Rejection Fine-Tuning (RFT), which enables the continuous co-evolution of data and models in a fully autonomous loop. The second stage introduces Group Relative Self-Distillation (GRSD), which identifies critical fork points in group rollouts and constructs dense step-level supervision from successful trajectories to correct failed ones. Extensive experiments on AndroidWorld show that our 4B model achieves an 81.0% Pass@1 success rate, outperforming numerous recent baselines and exceeding human-level performance. Ablation and case studies further verify the effectiveness of GRSD. Our method represents a significant leap toward efficient, self-evolving, and high-performance mobile GUI automation without expensive manual data annotation.