ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices
ProactiveMobile is a comprehensive benchmark for enhancing proactive intelligence on mobile devices, achieving a 20.82% success rate with Qwen2.5-VL-7B-Instruct.
Key Findings
Methodology
ProactiveMobile formalizes the proactive task by analyzing four dimensions of on-device contextual signals to infer latent user intent and generate executable function sequences from a pool of 63 APIs. The benchmark includes 3,660 instances across 14 scenarios, using multi-answer annotations to reflect real-world complexity.
Key Results
- Qwen2.5-VL-7B-Instruct achieved a 20.82% success rate in exact function sequence matching, significantly outperforming o1's 17.02% and GPT-5's 11.37%.
- Experiments indicate that current MLLMs generally lack proactivity but can improve with specialized training.
- The benchmark highlights the challenges of proactive intelligence, emphasizing the importance of ProactiveMobile.
Significance
ProactiveMobile provides a systematic evaluation framework for proactive intelligence research, addressing the oversimplification of existing benchmarks. By incorporating multi-answer annotations and generating executable function sequences, it offers researchers a more challenging and realistic testing environment, aiding the transition of mobile agents from passive executors to proactive collaborators.
Technical Contribution
ProactiveMobile introduces a novel task formalization by incorporating multi-dimensional contextual signals and executable function sequences. Compared to existing methods, it emphasizes multi-answer annotations and functional correctness, advancing technical progress in proactive intelligence.
Novelty
ProactiveMobile is the first to formalize the proactive task as inferring latent user intent across multi-dimensional contextual signals and generating executable sequences from a pool of 63 APIs, overcoming the single-answer limitation of existing benchmarks.
Limitations
- Current models still fall short of deployment requirements on devices, indicating the challenges of achieving proactive intelligence.
- The complexity of the benchmark may cause difficulties for models in handling multi-answer annotations.
Future Work
Future research could explore more efficient training methods to enhance proactive intelligence performance. Additionally, expanding the benchmark's scenarios and API pool could cover a broader range of applications.
AI Executive Summary
ProactiveMobile is a comprehensive benchmark designed to enhance proactive intelligence on mobile devices. Existing multimodal large language models (MLLMs) primarily operate within a reactive paradigm, limiting their seamless integration into daily life. ProactiveMobile introduces a novel task formalization by incorporating multi-dimensional contextual signals and executable function sequences, addressing the oversimplification of existing benchmarks.
The benchmark includes 3,660 instances across 14 scenarios, using multi-answer annotations to reflect real-world complexity. Experimental results show that the fine-tuned Qwen2.5-VL-7B-Instruct achieved a 20.82% success rate in exact function sequence matching, significantly outperforming o1 and GPT-5. This indicates that proactivity is a specialized capability requiring targeted training, highlighting the importance of ProactiveMobile in evaluating proactivity.
Despite significant progress, current models still fall short of deployment requirements on devices, indicating the challenges of achieving proactive intelligence. Future research could explore more efficient training methods and expand the benchmark's scenarios and API pool to cover a broader range of applications.
Deep Analysis
Background
In recent years, multimodal large language models (MLLMs) have made significant progress in mobile agent development. However, these models are primarily confined to a reactive paradigm, executing explicit user commands. This limitation hinders seamless integration into daily life, driving a shift towards proactive intelligence. The core of proactive intelligence is the ability of agents to autonomously anticipate user needs and initiate actions, reducing cognitive burden on users.
Core Problem
Existing benchmarks are overly simplistic, often assuming a single 'correct' action per scenario. This ignores the diversity and subjectivity of user preferences, forcing complex one-to-many mappings into an unrealistic one-to-one paradigm. Additionally, existing evaluation metrics fail to differentiate partial from complete failures, and output formats lack executability.
Innovation
ProactiveMobile introduces a novel task formalization by incorporating multi-dimensional contextual signals and executable function sequences. The benchmark emphasizes multi-answer annotations and functional correctness, allowing each instance to have one to three target actions. This approach transforms evaluation from a subjective text-matching problem into an objective, structured task.
Methodology
- �� Analyze four dimensions of on-device contextual signals: user profile, device status, world information, and behavioral trajectories.
- �� Generate executable function sequences from a pool of 63 APIs.
- �� Multi-answer annotations: each instance has one to three target actions.
- �� Three-stage review mechanism: ensures data reliability and accuracy.
Experiments
Experiments used the ProactiveMobile benchmark to evaluate the performance of various models, including fine-tuned Qwen2.5-VL-7B-Instruct and MiMo-VL-7B-SFT-2508. Results showed that Qwen2.5-VL-7B-Instruct achieved a 20.82% success rate in exact function sequence matching, significantly outperforming other models.
Results
Results indicated that the fine-tuned Qwen2.5-VL-7B-Instruct achieved a 20.82% success rate in exact function sequence matching, significantly outperforming o1 and GPT-5. This suggests that proactivity is a specialized capability requiring targeted training, emphasizing the importance of ProactiveMobile in evaluating proactivity.
Applications
ProactiveMobile can be used to evaluate and enhance proactive intelligence on mobile devices, applicable in scenarios requiring autonomous anticipation of user needs and actions, such as smart homes and personal assistants.
Limitations & Outlook
Current models still fall short of deployment requirements on devices, indicating the challenges of achieving proactive intelligence. Additionally, the complexity of the benchmark may cause difficulties for models in handling multi-answer annotations.
Plain Language Accessible to non-experts
Imagine having a super-smart assistant that not only listens to your commands but also knows what you need in advance. For example, when you walk into the kitchen, it already has your coffee ready because it knows you drink coffee every morning. This is how ProactiveMobile works. It analyzes your habits and environment to predict your needs and then takes action. This proactivity makes life easier because you no longer have to tell the assistant what to do every time.
ELI14 Explained like you're 14
Imagine you have a super-smart phone assistant. It not only understands what you say but also knows what you want in advance. For example, when you come home from school, it has already opened your favorite game because it knows you play it every day at this time. This assistant is like an invisible friend, always there to help you when you need it. ProactiveMobile is a system that makes phone assistants smarter and more understanding.
Glossary
Multimodal Large Language Models (MLLMs)
Language models that combine multiple data types (e.g., text, images) to understand and generate natural language.
Used in mobile agent development to enhance their ability to understand and execute complex tasks.
Proactive Intelligence
The ability of a system to autonomously anticipate user needs and take action.
Achieved in ProactiveMobile through multi-dimensional contextual signals.
Function Pool
A set of predefined executable functions used to translate user intent into specific operations.
ProactiveMobile uses a pool of 63 APIs to generate executable sequences.
Multi-Answer Annotations
An annotation method that allows multiple correct answers per instance, reflecting the diversity of user preferences.
Used in ProactiveMobile benchmark for instance annotation.
Exact Function Sequence Matching
The model's output function sequence exactly matches the target sequence in the benchmark.
Used to evaluate model performance on ProactiveMobile.
Open Questions Unanswered questions from this research
- 1 How to improve model handling of multi-answer annotations remains an open question.
- 2 Current models fall short in deployment performance on devices, requiring exploration of more efficient training methods.
Applications
Immediate Applications
Smart Homes
By predicting users' daily habits, automatically adjust home devices to enhance convenience.
Long-term Vision
Personal Assistants
Develop personal assistants capable of autonomously handling complex tasks, reducing cognitive load on users.
Abstract
Multimodal large language models (MLLMs) have made significant progress in mobile agent development, yet their capabilities are predominantly confined to a reactive paradigm, where they merely execute explicit user commands. The emerging paradigm of proactive intelligence, where agents autonomously anticipate needs and initiate actions, represents the next frontier for mobile agents. However, its development is critically bottlenecked by the lack of benchmarks that can address real-world complexity and enable objective, executable evaluation. To overcome these challenges, we introduce ProactiveMobile, a comprehensive benchmark designed to systematically advance research in this domain. ProactiveMobile formalizes the proactive task as inferring latent user intent across four dimensions of on-device contextual signals and generating an executable function sequence from a comprehensive function pool of 63 APIs. The benchmark features over 3,660 instances of 14 scenarios that embrace real-world complexity through multi-answer annotations. To ensure quality, a team of 30 experts conducts a final audit of the benchmark, verifying factual accuracy, logical consistency, and action feasibility, and correcting any non-compliant entries. Extensive experiments demonstrate that our fine-tuned Qwen2.5-VL-7B-Instruct achieves a success rate of 19.15%, outperforming o1 (15.71%) and GPT-5 (7.39%). This result indicates that proactivity is a critical competency widely lacking in current MLLMs, yet it is learnable, emphasizing the importance of the proposed benchmark for proactivity evaluation.