PhoneWorld: Scaling Phone-Use Agent Environments
PhoneWorld converts real GUI trajectories into scalable phone-use environments, boosting evaluation benchmarks by 52.5 points.
Key Findings
Methodology
PhoneWorld uses a reusable pipeline to convert real GUI trajectories and screenshots into controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts. It recovers screen connections and user goals to build mock Android apps and derive tasks and verifiers.
Key Results
- Replacing 10K steps with PhoneWorld supervision improves HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points.
- Increasing PhoneWorld supervision significantly enhances performance, and expanding app coverage yields larger gains under fixed budgets.
- RL case study shows PhoneWorld's automatic verification rewards improve real-phone task success by 5.66%.
Significance
PhoneWorld shifts focus from building benchmarks one at a time to scaling verifiable phone-use environments. It offers new possibilities for evaluation, supervision, and reinforcement learning, addressing the long-standing issue of environment supply.
Technical Contribution
PhoneWorld provides a scalable method for environment construction, supporting both evaluation and training. Its automatic verification mechanism offers new reward designs for reinforcement learning.
Novelty
PhoneWorld is the first to construct large-scale phone-use environments using real trajectories, focusing on scalability and verifiability, unlike existing methods.
Limitations
- PhoneWorld relies on the quality and coverage of real trajectories, which may not fully simulate complex app interactions.
- Environment construction requires significant initial data collection and human auditing, limiting automation.
Future Work
Future work includes expanding app coverage, optimizing the automation process, and exploring cross-platform environment construction.
AI Executive Summary
A major bottleneck for phone-use agents is the difficulty in constructing scalable, controllable environments. Existing benchmarks have made progress in evaluation but lack scalable methods for expanding phone-use environments. PhoneWorld addresses this by converting real GUI trajectories and screenshots into controllable environments. It supports both evaluation and training, improving multiple evaluation benchmarks. PhoneWorld's automatic verification mechanism provides new reward designs for reinforcement learning, showing potential in real-phone tasks. However, PhoneWorld still faces challenges in automation and coverage. Future work will focus on expanding app coverage and optimizing the construction process.
Deep Analysis
Background
Phone-use agents need to handle visually rich, stateful interfaces. Existing benchmarks focus on evaluation rather than environment construction. PhoneWorld creates controllable environments using real trajectories and screenshots, addressing the issue of environment supply.
Core Problem
The core problem for phone-use agents is the lack of environment supply, making it difficult to construct scalable, controllable environments, limiting model training and evaluation.
Innovation
PhoneWorld recovers screen connections and user goals from real trajectories, building mock Android apps and deriving tasks and verifiers. This method supports both evaluation and training.
Methodology
- �� Recover screen connections and user goals
- �� Build mock Android apps
- �� Derive executable tasks and verifiers
- �� Generate training rollouts
Experiments
The experimental design includes replacing 10K steps in AndroidWorld with PhoneWorld supervision, evaluating its impact on multiple benchmarks, and conducting an RL case study.
Results
Replacing 10K steps with PhoneWorld supervision improves HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points.
Applications
PhoneWorld can be used for evaluation, supervision, and reinforcement learning, suitable for research requiring scalable environment construction for phone-use agents.
Limitations & Outlook
PhoneWorld relies on the quality and coverage of real trajectories, and environment construction requires significant initial data collection and human auditing.
Plain Language Accessible to non-experts
Imagine you're in a huge app store, where each app has its own booth. PhoneWorld is like a smart guide that quickly sets up these booths, allowing you to easily browse and interact. It not only shows the app's appearance but also simulates user operations, helping you understand how the app is used. In this way, PhoneWorld provides a controllable training and evaluation environment for phone-use agents.
ELI14 Explained like you're 14
Imagine you're playing a mobile game with many levels, each level is a different app. PhoneWorld is like a super game editor that quickly creates these levels, allowing you to explore freely. It not only shows the app's appearance but also simulates user operations, helping you understand how the app is used. This way, you can easily learn how to use various apps in the game!
Glossary
PhoneWorld
A reusable pipeline for creating controllable phone-use environments.
Used to convert real GUI trajectories into controllable environments.
GUI trajectories
The screen paths users take within an app.
Used to recover screen connections and user goals.
Automatic verifier
A program that automatically checks task completion.
Used to verify if user goals are achieved.
Training rollouts
Data records used for model training.
Generated from successful operations in controllable environments.
Mock Android app
A runnable app constructed from real trajectories.
Provides a controllable phone-use environment.
Open Questions Unanswered questions from this research
- 1 How to further automate the environment construction process?
- 2 How to expand PhoneWorld's app coverage?
- 3 How to improve the quality and coverage of real trajectories?
Applications
Immediate Applications
Phone Agent Evaluation
PhoneWorld provides a controllable environment for evaluating phone agent performance.
Long-term Vision
Cross-Platform Environment Construction
Explore PhoneWorld's application across different platforms, providing a unified environment construction solution.
Abstract
A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Existing mobile-agent benchmarks have made important progress on evaluation, but they do not by themselves provide a scalable way to construct many new phone-use environments. We present PhoneWorld, a reusable pipeline that converts real GUI trajectories and screenshots into controllable phone-use environments, executable tasks, automatic verifiers, and training rollouts. Rather than hand-building one mobile benchmark at a time, PhoneWorld uses real trajectories to recover which screens matter, how screens connect, which interactions must change environment state, and which user goals admit automatic verification. From these signals, it builds runnable mock Android apps backed by read-only app content and mutable state, then derives executable tasks, rule-based verifiers, and training rollouts from the same environments. In its current instantiation, PhoneWorld covers 34 apps across 16 domains, spanning common consumer mobile behaviors such as search, browsing, shopping, booking, media, and social interaction. Under a fixed training budget, replacing 10K steps from an auxiliary AndroidWorld corpus in an AndroidWorld-based baseline with broad PhoneWorld supervision improves all four evaluation benchmarks at once, raising HYMobileBench by 17.7 points, AndroidControl by 6.0 points, AndroidWorld by 14.7 points, and PhoneWorld by 52.5 points. We then study two additional scaling questions: increasing the amount of PhoneWorld supervision strongly improves PhoneWorld performance, and under a fixed PhoneWorld budget, expanding app coverage yields even larger gains. Overall, PhoneWorld shifts the focus from building one mobile benchmark at a time to scaling the supply of phone-use environments themselves.