MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning
Proposed MobileIPL with iterative preference learning significantly improves mobile GUI reasoning, outperforming SOTA models by over 10% on key benchmarks.
Key Findings
Methodology
This paper introduces an iterative sampling framework based on CoaT trees, utilizing Monte Carlo Tree Search (MCTS) to build diverse reasoning paths. Leaf nodes are scored via rule-based rewards, and backward propagation generates thinking-level DPO pairs. Coupled with GPT-4o-driven instruction evolution, the approach enriches training data, mitigates overfitting, and enhances reasoning diversity. Multi-turn dialogue with vision-language models (VLMs) incrementally constructs the CoaT-tree, evaluating intermediate reasoning steps without costly process-level annotations. The method effectively balances exploration and exploitation, leading to improved generalization and reasoning quality in mobile GUI tasks.
Key Results
- On the AITZ dataset, MobileIPL achieves 91.73% Action Type match, surpassing baseline models by over 10%. In AMEX, it improves performance by 3.58%, reaching 70.93%. On AndroidControl, Step.Acc reaches 72.7%, outperforming Falcon-UI, demonstrating strong out-of-domain generalization. The iterative T-DPO training enhances reasoning path quality, reducing reward hacking and increasing robustness.
- Instruction evolution generates diverse QA pairs grounded in real UI screenshots, broadening reasoning contexts and layout understanding. This strategy improves sample diversity and model robustness, especially under limited data conditions. Ablation studies confirm that both IPL and instruction evolution contribute significantly to performance gains.
- The combination of rule-based rewards, hierarchical reasoning, and multi-round sampling leads to more accurate and diverse reasoning paths. The method reduces reliance on expensive annotations, offering a scalable solution for complex multi-modal tasks, and demonstrates superior performance across multiple benchmarks.
Significance
This work addresses the critical bottleneck of sparse intermediate rewards in GUI reasoning tasks. By introducing an efficient, annotation-free iterative preference learning framework, it enhances the reasoning capacity and generalization of mobile agents. The hierarchical CoaT-tree structure and multi-path sampling provide a new paradigm for integrating structured reasoning with preference optimization. The approach's ability to outperform existing models with less data and computational cost signifies a breakthrough in scalable, autonomous mobile AI. Its potential applications span automated app testing, intelligent assistants, and human-computer interaction, promising to elevate the intelligence and autonomy of future mobile systems.
Technical Contribution
The paper pioneers a hierarchical CoaT-tree-based sampling mechanism combined with rule-based reward scoring, eliminating the need for costly process annotations. It innovatively integrates GPT-4o-driven instruction evolution to diversify training data, thereby improving reasoning robustness. The multi-round T-DPO training framework refines the agent’s reasoning policy iteratively, significantly outperforming existing continual pretraining models. The method’s design balances exploration and exploitation, enabling efficient learning in complex, multimodal GUI environments, and sets a new standard for preference-based reasoning in mobile agents.
Novelty
This is the first work to leverage CoaT trees with Monte Carlo sampling for GUI reasoning, avoiding expensive intermediate annotations. The integration of rule-based rewards with hierarchical backpropagation for preference optimization is novel. Additionally, GPT-4o-driven instruction evolution introduces a new way to generate diverse training data, addressing overfitting and limited sample diversity issues. These innovations collectively push the boundary of scalable, preference-guided reasoning in multimodal mobile AI, distinguishing this approach from prior end-to-end supervised or reward-model-dependent methods.
Limitations
- The reliance on rule-based rewards may limit the granularity of intermediate reasoning evaluation, especially in complex or ambiguous scenarios, potentially affecting reasoning accuracy.
- Hierarchical sampling and tree construction incur significant computational overhead, which may hinder real-time deployment in resource-constrained environments.
- Dependence on GPT-4o for instruction evolution could restrict applicability in settings with limited access to large language models, and the method's scalability to more complex tasks remains to be validated.
Future Work
Future directions include developing learned reward models to replace rule-based scoring, improving sampling efficiency through reinforcement learning techniques, and extending the framework to handle more complex, multi-modal reasoning tasks. Additionally, exploring lightweight instruction evolution strategies and integrating more advanced vision-language models could further enhance generalization and real-time performance. The goal is to build more autonomous, adaptable mobile agents capable of reasoning in diverse, real-world scenarios with minimal supervision.
AI Executive Summary
In recent years, mobile graphical user interface (GUI) agents have gained attention for their potential to automate complex tasks on smartphones and tablets. Existing approaches largely depend on supervised fine-tuning with trajectory data, which often leads to overfitting and limited reasoning diversity. To overcome these challenges, this paper introduces MobileIPL, a novel framework that leverages iterative preference learning (IPL) to enhance the reasoning process of mobile agents.
The core innovation lies in constructing a hierarchical CoaT-tree through Monte Carlo Tree Search (MCTS), where each node represents a reasoning step. Instead of relying on expensive process-level annotations, the method scores leaf nodes using rule-based rewards, then propagates these scores backward to assign credit to intermediate reasoning steps. This process generates thinking-level DPO pairs, which are used to fine-tune the agent via direct preference optimization. To further improve diversity and prevent overfitting, the authors employ a three-stage instruction evolution strategy powered by GPT-4o, generating varied question-answer pairs grounded in real mobile UI screenshots.
Extensive experiments on three benchmark datasets—AITZ, AMEX, and AndroidControl—demonstrate that MobileIPL outperforms strong baselines such as OS-ATLAS and UI-TARS. Notably, it achieves state-of-the-art results, with Action Type match rates exceeding 91% and Step.Acc reaching 72.7%. The iterative T-DPO training significantly enhances reasoning quality, robustness, and out-of-domain generalization. Ablation studies confirm the critical roles of IPL and instruction evolution in achieving these gains.
This work advances the field by providing a scalable, annotation-free approach to hierarchical reasoning and preference optimization in multimodal GUI tasks. Its implications extend to automating app testing, improving virtual assistants, and enabling more autonomous mobile AI systems. Future efforts will focus on integrating learned reward models, reducing computational costs, and extending to more complex, multi-modal reasoning scenarios, paving the way for truly intelligent mobile agents.
Deep Dive
Abstract
The Chain of Action-Planning Thoughts (CoaT) paradigm has been shown to improve the reasoning performance of VLM-based mobile agents in GUI tasks. However, the scarcity of diverse CoaT trajectories limits the expressiveness and generalization ability of such agents. While self-training is commonly employed to address data scarcity, existing approaches either overlook the correctness of intermediate reasoning steps or depend on expensive process-level annotations to construct process reward models (PRM). To address the above problems, we propose an Iterative Preference Learning (IPL) that constructs a CoaT-tree through interative sampling, scores leaf nodes using rule-based reward, and backpropagates feedback to derive Thinking-level Direct Preference Optimization (T-DPO) pairs. To prevent overfitting during warm-up supervised fine-tuning, we further introduce a three-stage instruction evolution, which leverages GPT-4o to generate diverse Q\&A pairs based on real mobile UI screenshots, enhancing both generality and layout understanding. Experiments on three standard Mobile GUI-agent benchmarks demonstrate that our agent MobileIPL outperforms strong baselines, including continual pretraining models such as OS-ATLAS and UI-TARS. It achieves state-of-the-art performance across three standard Mobile GUI-Agents benchmarks and shows strong generalization to out-of-domain scenarios.