Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-based Mobile-Using Agents
Mobile-Aptus enhances mobile agents' interaction via confidence-driven framework, boosting task success rate by 26%.
Key Findings
Methodology
Mobile-Aptus employs a two-stage framework: interaction capability empowerment and confidence bias correction. The first stage involves supervised fine-tuning to learn actions and confidence scores. The second stage combines semantic similarity retrieval with Direct Preference Optimization (DPO) to improve the accuracy of confidence scores.
Key Results
- On benchmarks like OS-Kairos, AITZ, Meta-GUI, and AndroidControl, Mobile-Aptus improved task success rates by an average of 17%.
- In dynamic experiments, it surpassed the baseline by 26% in task success rate with only 0.64 intervention steps per instruction.
- Replacing human interventions with a multi-agent system led to a 32.91% improvement in task success rate.
Significance
This research significantly enhances the autonomy and robustness of mobile agents in multimodal large language models, addressing the issues of over-execution and over-soliciting human intervention, providing a more efficient solution for smartphone automation.
Technical Contribution
Introduces a universal confidence integration framework combining semantic similarity retrieval and DPO, overcoming existing methods' deficiencies in confidence judgment, offering new theoretical guarantees and engineering possibilities.
Novelty
First to introduce confidence-driven interaction in multimodal large language model-based mobile agents, significantly reducing unnecessary human interventions.
Limitations
- In complex tasks, the accuracy of confidence scores still needs improvement, potentially leading to misjudgments.
- The model's generalization ability in specific application scenarios is limited.
Future Work
Future research could explore more complex confidence scoring mechanisms and validate its effectiveness in more application scenarios.
AI Executive Summary
Recent advancements in multimodal large language models (MLLMs) have significantly improved visual perception, task planning, and reasoning capabilities, laying a strong foundation for mobile agents. However, fully automated agents often attempt tasks they cannot resolve, leading to over-execution.
Mobile-Aptus introduces a confidence-driven interaction framework that addresses both over-execution and over-soliciting human intervention. Experimental results show state-of-the-art performance across multiple benchmarks, significantly improving task success rates.
This study holds significant academic value and offers a more efficient solution for smartphone automation. However, the accuracy of confidence scores in complex tasks needs improvement, and future research should explore more complex confidence scoring mechanisms.
Deep Analysis
Background
Multimodal large language models (MLLMs) have recently made significant strides in visual perception, task planning, and reasoning, advancing the development of mobile agents. Early mobile agents were primarily script or rule-driven, with limited flexibility. Advances in MLLMs offer new possibilities for developing general mobile agents based on graphical user interfaces.
Core Problem
Fully automated mobile agents often attempt tasks they cannot resolve, leading to over-execution. Additionally, existing interactive agents excessively rely on human intervention, increasing task execution time and user burden.
Innovation
Mobile-Aptus addresses over-execution and over-soliciting human intervention through a confidence-driven interaction framework. Its core innovation lies in combining semantic similarity retrieval and Direct Preference Optimization (DPO) to improve the accuracy of confidence scores.
Methodology
- �� Interaction capability empowerment stage: Learn actions and confidence scores through supervised fine-tuning.
- �� Confidence bias correction stage: Improve confidence score accuracy using semantic similarity retrieval and DPO.
- �� Train and test using the OS-Kairos dataset.
Experiments
Experiments were conducted on benchmarks like OS-Kairos, AITZ, Meta-GUI, and AndroidControl, evaluating metrics such as task success rate and step-wise success rate. Baselines included various open-source and closed-source mobile agent models.
Results
Mobile-Aptus achieved state-of-the-art performance across multiple benchmarks, with an average task success rate improvement of 17%. In dynamic experiments, it surpassed the baseline by 26% in task success rate with only 0.64 intervention steps per instruction.
Applications
This method can be applied to automate tasks on smartphones, reducing user intervention and increasing efficiency. It is suitable for applications requiring high autonomy and robustness.
Limitations & Outlook
The accuracy of confidence scores in complex tasks still needs improvement, potentially leading to misjudgments. Additionally, the model's generalization ability in specific application scenarios is limited.
Plain Language Accessible to non-experts
Imagine a smartphone assistant that helps you with various tasks like opening apps and sending messages. This assistant is like a smart friend who can usually handle tasks independently but sometimes asks if you want help. Mobile-Aptus makes this assistant smarter by better judging when it needs your help and when it can handle things on its own. This way, you don't have to constantly watch your phone, and the assistant won't bother you unnecessarily.
ELI14 Explained like you're 14
Imagine you have a super-smart phone assistant that helps you do lots of things, like opening apps and sending messages. It's like a game character that can usually handle tasks on its own but sometimes asks if you need help. Mobile-Aptus makes this assistant even smarter by helping it decide when it really needs your help and when it can manage on its own. This way, you can use your phone more easily without being interrupted all the time!
Glossary
Multimodal Large Language Model (MLLM)
Language models that combine multiple data modalities (e.g., text, images) to enhance understanding and generation capabilities.
Used to enhance mobile agents' task execution capabilities.
Confidence Scoring
A quantitative representation of a model's confidence in its output.
Used to determine whether human intervention is needed.
Direct Preference Optimization (DPO)
An optimization strategy that improves the accuracy of model outputs through preference learning.
Used to correct biases in confidence scoring.
Semantic Similarity Retrieval
A technique for retrieving relevant information by calculating semantic similarity.
Used to improve the accuracy of confidence scores.
Interaction Capability Empowerment
Enhancing a model's ability to interact with its environment through learning.
First stage of Mobile-Aptus.
Open Questions Unanswered questions from this research
- 1 How to improve confidence score accuracy in more complex tasks?
- 2 Validate the model's effectiveness in more application scenarios.
Applications
Immediate Applications
Smartphone Automation
Reduce user intervention and increase task execution efficiency. Suitable for applications requiring high autonomy and robustness.
Long-term Vision
Evolution of Smart Assistants
Enhance user experience through smarter decision-making and interaction, reducing unnecessary interventions.
Abstract
Recent advancements in multimodal large language models (MLLMs) have shown exceptional potential in enabling mobile-using agents to autonomously execute human instructions. However, fully automated agents often try to execute tasks even when they are unable to resolve them, leading to the problem of over-execution. Previous studies solve it by training a interactive mobile-using agents to let agents request human interaction when agents can not complete user instructions. However, we find that these interactive agents tend to exhibit over-soliciting behavior, relying excessively on human intervention. To mitigate both over-execution and over-soliciting, we propose a universal confidence integration framework that enables confidence-driven proactive and robust interaction in MLLM-based mobile-using agents. The framework consists of two stages: interaction capability empowerment and confidence bias correction. In the interaction capability empowerment stage, agents learn through supervised fine-tuning to output both actions and confidence scores. In the confidence bias correction stage, agents learn to output more accurate confidence scores by combining semantic similarity retrieval with direct preference optimization. Experimental results show Mobile-Aptus achieves state-of-the-art performance on the four popular mobile-using agent benchmarks: OS-Kairos, AITZ, Meta-GUI, and AndroidControl. Mobile-Aptus consistently outperforms all baselines in offline benchmarks, with an average improvement over 17\% in task success rate. In real-world dynamic experiments, Mobile-Aptus surpasses the baseline by 26% in task success rate with only 0.64 intervention steps per instruction. The codes are available at https://github.com/Wuzheng02/Mobile-Aptus.