UI-Venus-1.5 Technical Report
UI-Venus-1.5 achieves 77.6% success rate in AndroidWorld navigation via model merging and online reinforcement learning.
Key Findings
Methodology
UI-Venus-1.5 employs three key techniques: Mid-Training with 10 billion tokens for GUI semantics; Online Reinforcement Learning with full trajectory rollouts; Model Merging to synthesize domain-specific models into one unified checkpoint.
Key Results
- Achieved 69.6% accuracy on ScreenSpot-Pro, surpassing baselines like Seed1.8 and Holo2.
- Reached 75.0% accuracy on VenusBench-GD, significantly outperforming MAI-UI.
- Attained a 77.6% success rate on AndroidWorld, exceeding Mobile-Agent-v3.
Significance
UI-Venus-1.5 has a significant impact on academia and industry, addressing long-standing pain points in large-scale navigation and enhancing the practicality of GUI agents.
Technical Contribution
UI-Venus-1.5 offers new theoretical guarantees and engineering possibilities through model merging and online reinforcement learning, distinctly different from existing SOTA methods.
Novelty
This is the first to combine Mid-Training with Model Merging for GUI agents, addressing the domain shift between training data and real-world environments.
Limitations
- Navigation performance may drop in complex mobile apps due to dynamic environment changes.
- Model merging may cause slight performance loss in individual domains.
Future Work
Future directions include extending to more language environments and optimizing model merging strategies to improve cross-domain performance.
AI Executive Summary
UI-Venus-1.5 is a unified GUI agent designed to tackle automation challenges in digital environments. Traditional methods often rely on rigid APIs, while UI-Venus-1.5 interacts directly with graphical interfaces through visual perception, mimicking human behavior for navigation. The model employs a Mid-Training stage, utilizing 10 billion tokens to establish a solid GUI semantic foundation. Through online reinforcement learning and model merging, UI-Venus-1.5 achieves breakthrough performance across multiple benchmarks, notably reaching a 77.6% success rate on AndroidWorld. Despite these achievements, the model faces challenges in certain dynamic environments, and future work will focus on further optimizing model merging strategies.
Deep Analysis
Background
With the rapid evolution of multimodal large language models, GUI agents have emerged as a bridge between human instructions and digital execution. Traditional automation tools rely on rigid APIs, while GUI agents interact directly with graphical interfaces through visual perception, mimicking human behavior for navigation.
Core Problem
Creating intelligent systems capable of autonomously operating digital devices has long been a central goal in AI. Existing methods struggle to balance broad generality and strong task performance.
Innovation
UI-Venus-1.5 introduces Mid-Training, Online Reinforcement Learning, and Model Merging. These innovations enable the model to perform long-horizon navigation in large-scale environments and synthesize domain-specific models into one cohesive checkpoint.
Methodology
- �� Mid-Training: Utilizes 10 billion tokens to establish GUI semantic foundation.
- �� Online Reinforcement Learning: Optimizes long-horizon navigation through full trajectory rollouts.
- �� Model Merging: Synthesizes domain-specific models into one unified checkpoint.
Experiments
Experimental design includes evaluating model performance on benchmarks like ScreenSpot-Pro, VenusBench-GD, and AndroidWorld. Utilizes various datasets and reinforcement learning strategies for training.
Results
UI-Venus-1.5 achieved 69.6% accuracy on ScreenSpot-Pro, 75.0% on VenusBench-GD, and a 77.6% success rate on AndroidWorld.
Applications
UI-Venus-1.5 demonstrates robust navigation capabilities across various Chinese mobile apps, effectively executing user instructions in real-world scenarios such as ticket booking, purchase of goods, and automated conversation management.
Limitations & Outlook
Despite UI-Venus-1.5's strong performance across benchmarks, it faces challenges in certain dynamic environments. Future work will focus on optimizing model merging strategies.
Plain Language Accessible to non-experts
Imagine you are shopping in a large supermarket. UI-Venus-1.5 acts like a smart shopping assistant that understands your shopping list and helps you find all the items you need. It not only recognizes the types of products but also navigates the store based on your instructions to find the best route. It's like a smart navigation system that understands your needs and helps you quickly find your target in a complex environment.
ELI14 Explained like you're 14
Imagine you're playing a super complex game where you need to find hidden treasure on a huge map. UI-Venus-1.5 is like your super assistant, understanding your instructions and helping you find the treasure's location. It not only recognizes the map's markers but also navigates the game based on your instructions to find the best route. It's like a smart navigation system that understands your needs and helps you quickly find your target in a complex environment.
Glossary
GUI Agent
Graphical User Interface agent that interacts with interfaces through visual perception.
Used in UI-Venus-1.5 for automated interactions.
Reinforcement Learning
Optimizes policies through reward mechanisms.
Used to enhance UI-Venus-1.5's navigation capabilities.
Model Merging
Combines multiple domain models into one unified model.
Used to create a single GUI agent.
Mid-Training
Training phase before reinforcement learning.
Used to establish GUI semantic foundation.
Trajectory Rollout
Optimization process of complete navigation paths.
Used in the online reinforcement learning stage.
Open Questions Unanswered questions from this research
- 1 How to further improve navigation performance in dynamic environments?
- 2 How to optimize the impact of model merging on individual domain performance?
Applications
Immediate Applications
Mobile App Navigation
UI-Venus-1.5 can be used for complex mobile app navigation, helping users quickly find target functions.
Long-term Vision
Smart Assistant
UI-Venus-1.5 has the potential to become a smart assistant in everyday life, helping users efficiently complete tasks in digital environments.
Abstract
GUI agents have emerged as a powerful paradigm for automating interactions in digital environments, yet achieving both broad generality and consistently strong task performance remains challenging. In this report, we present UI-Venus-1.5, a unified, end-to-end GUI Agent designed for robust real-world applications. The proposed model family comprises two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B) to meet various downstream application scenarios. Compared to our previous version, UI-Venus-1.5 introduces three key technical advances: (1) a comprehensive Mid-Training stage leveraging 10 billion tokens across 30+ datasets to establish foundational GUI semantics; (2) Online Reinforcement Learning with full-trajectory rollouts, aligning training objectives with long-horizon, dynamic navigation in large-scale environments; and (3) a single unified GUI Agent constructed via Model Merging, which synthesizes domain-specific models (grounding, web, and mobile) into one cohesive checkpoint. Extensive evaluations demonstrate that UI-Venus-1.5 establishes new state-of-the-art performance on benchmarks such as ScreenSpot-Pro (69.6%), VenusBench-GD (75.0%), and AndroidWorld (77.6%), significantly outperforming previous strong baselines. In addition, UI-Venus-1.5 demonstrates robust navigation capabilities across a variety of Chinese mobile apps, effectively executing user instructions in real-world scenarios. Code: https://github.com/inclusionAI/UI-Venus; Model: https://huggingface.co/collections/inclusionAI/ui-venus