GptGet

UI-Venus Technical Report: Building High-performance UI Agents with RFT

TL;DR

UI-Venus achieves SOTA in UI grounding/navigation using RFT.

cs.CV 🔴 Advanced 2025-08-15 6 views
Zhangxuan Gu Zhengwen Zeng Zhenyu Xu Xingran Zhou Shuheng Shen Yunfei Liu Beitong Zhou Changhua Meng Tianyu Xia Weizhi Chen Yue Wen Jingya Dou Fei Tang Jinzhen Lin Yulin Liu Zhenlin Guo Yichen Gong Heng Jia Changlong Gao Yuan Guo Yong Deng Zhenyu Guo Liang Chen Weiqiang Wang
AI Reader Arxiv Page Download PDF
UI agents reinforcement learning multimodal model data cleaning self-evolving

Key Findings

Methodology

UI-Venus is based on Qwen2.5-VL, utilizing reinforcement fine-tuning (RFT) for efficient UI grounding and navigation. It employs a Self-Evolving Trajectory History Alignment and Sparse Action Enhancement framework to boost navigation performance. Detailed reward functions and data cleaning strategies are designed.

Key Results

  • UI-Venus achieves 94.1% and 95.3% accuracy on Screenspot-V2/Pro, surpassing GTA1 and UI-TARS-1.5.
  • Achieves 49.1% and 65.9% success rates on AndroidWorld, outperforming existing models.
  • The self-evolving framework significantly improves planning coherence and generalization in complex UI tasks.

Significance

UI-Venus represents a breakthrough in UI agent technology, particularly in UI grounding and navigation tasks. By introducing a self-evolving framework and efficient data processing methods, it addresses long-standing issues of data quality and generalization, providing new research directions for academia and industry.

Technical Contribution

UI-Venus introduces a self-evolving framework and sparse action enhancement, significantly improving navigation task performance. Compared to SOTA methods, it achieves breakthroughs in data quality and generalization, offering new engineering possibilities.

Novelty

UI-Venus is the first to introduce a Self-Evolving Trajectory History Alignment and Sparse Action Enhancement framework in UI agent tasks, significantly improving planning coherence and generalization compared to existing methods.

Limitations

  • In complex dynamic environments, UI-Venus's planning capabilities still have room for improvement, especially in handling long-chain tasks.
  • The data cleaning process relies on manual intervention, which may affect large-scale application.

Future Work

Future research can explore automated data cleaning and more efficient self-evolving mechanisms to further enhance the performance and adaptability of UI agents.

AI Executive Summary

UI-Venus is a UI agent based on a multimodal large language model, focusing on UI grounding and navigation tasks. Through reinforcement fine-tuning and a self-evolving framework, UI-Venus achieves SOTA performance across multiple benchmarks. In UI grounding tasks, UI-Venus achieves 94.1% and 95.3% accuracy on Screenspot-V2 and Pro, surpassing existing models like GTA1 and UI-TARS-1.5. In UI navigation tasks, UI-Venus achieves 49.1% and 65.9% success rates on AndroidWorld, also outperforming current models. The success of UI-Venus is attributed to its innovative Self-Evolving Trajectory History Alignment and Sparse Action Enhancement framework, which significantly improves planning coherence and generalization in complex UI tasks. Despite this, UI-Venus still has room for improvement in handling complex dynamic environments and long-chain tasks. Future research can further explore automated data cleaning and more efficient self-evolving mechanisms.

Deep Analysis

Background

In recent years, multimodal large language models have made significant progress in the field of UI agents. Traditional methods rely on supervised fine-tuning with large datasets, which, while effective in some tasks, struggle with generalization in UI grounding tasks. Reinforcement fine-tuning (RFT) has emerged as a promising approach, significantly improving model generalization through reward functions and data cleaning strategies.

Core Problem

The core problem in UI agent tasks is achieving efficient UI grounding and navigation in complex, dynamic environments. Traditional methods perform poorly in handling long-chain tasks and dynamic environments, particularly facing bottlenecks in data quality and generalization.

Innovation

UI-Venus introduces a Self-Evolving Trajectory History Alignment and Sparse Action Enhancement framework, significantly improving planning coherence and generalization in navigation tasks. By designing detailed reward functions and efficient data cleaning strategies, UI-Venus achieves breakthroughs in data quality and generalization.

Methodology

  • �� Reinforcement fine-tuning using Qwen2.5-VL
  • �� Self-Evolving Trajectory History Alignment for improved planning coherence
  • �� Sparse Action Enhancement for optimizing sparse but critical action distribution
  • �� Three-stage data cleaning process to ensure high-quality training data

Experiments

Experiments were conducted using benchmarks like Screenspot-V2/Pro and AndroidWorld, comparing UI-Venus with models like GTA1 and UI-TARS-1.5. The effectiveness of the self-evolving framework and sparse action enhancement was validated through performance comparisons of different variants.

Results

UI-Venus achieves 94.1% and 95.3% accuracy on Screenspot-V2/Pro and 49.1% and 65.9% success rates on AndroidWorld, surpassing existing SOTA models.

Applications

UI-Venus can be applied in automated UI testing, user interface optimization, and intelligent assistant development, particularly in scenarios requiring high-precision UI grounding and complex navigation.

Limitations & Outlook

Despite its strong performance across multiple tasks, UI-Venus still has room for improvement in handling complex dynamic environments and long-chain tasks. Additionally, the manual dependency in the data cleaning process may limit its large-scale application.

Plain Language Accessible to non-experts

Imagine you're in a large shopping mall looking for a specific store. Traditional methods are like relying on maps and signs, while UI-Venus is like a smart shopping assistant. It not only knows the store's location but can also adjust the route in real-time based on your needs and the mall's dynamic changes. This smart assistant continuously learns and optimizes its navigation strategy, ensuring you quickly find the target store, even in a complex shopping environment.

ELI14 Explained like you're 14

Imagine you're playing a complex maze game, aiming to find hidden treasure. Traditional methods are like giving you a static map, while UI-Venus is like a super-smart game assistant. It not only knows the treasure's location but can also adjust the route in real-time based on maze changes, helping you avoid traps and dead ends. This assistant continuously learns and optimizes its strategy, ensuring you find the treasure fastest and win the game!

Glossary

Reinforcement Fine-Tune

A training method that optimizes model performance through reward mechanisms, particularly suitable for decision-making tasks.

Used in UI-Venus to enhance UI navigation task performance.

Self-Evolving Trajectory History Alignment

A method to optimize historical reasoning traces, aiming to improve model planning coherence.

Used in UI navigation tasks to optimize decision-making processes.

Sparse Action Enhancement

Optimizes the distribution of sparse but critical actions to improve model performance in complex tasks.

Enhances generalization in UI navigation tasks.

Qwen2.5-VL

A multimodal large language model supporting text and visual information processing.

Used as the base model for reinforcement fine-tuning in UI-Venus.

Screenspot-V2/Pro

Standard datasets for evaluating UI grounding task performance.

UI-Venus achieves SOTA performance on these datasets.

Open Questions Unanswered questions from this research

  • 1 How to achieve efficient data cleaning without manual intervention to support large-scale applications.
  • 2 How to further enhance UI agent planning and decision-making capabilities in complex dynamic environments.

Applications

Immediate Applications

Automated UI Testing

UI-Venus can enhance the efficiency and accuracy of software testing, reducing manual intervention.

Long-term Vision

Intelligent User Interface Assistant

Continuously learns user behavior and interface changes to provide personalized interface optimization suggestions.

Abstract

We present UI-Venus, a native UI agent that takes only screenshots as input based on a multimodal large language model. UI-Venus achieves SOTA performance on both UI grounding and navigation tasks using only several hundred thousand high-quality training samples through reinforcement finetune (RFT) based on Qwen2.5-VL. Specifically, the 7B and 72B variants of UI-Venus obtain 94.1% / 50.8% and 95.3% / 61.9% on the standard grounding benchmarks, i.e., Screenspot-V2 / Pro, surpassing the previous SOTA baselines including open-source GTA1 and closed-source UI-TARS-1.5. To show UI-Venus's summary and planing ability, we also evaluate it on the AndroidWorld, an online UI navigation arena, on which our 7B and 72B variants achieve 49.1% and 65.9% success rate, also beating existing models. To achieve this, we introduce carefully designed reward functions for both UI grounding and navigation tasks and corresponding efficient data cleaning strategies. To further boost navigation performance, we propose Self-Evolving Trajectory History Alignment & Sparse Action Enhancement that refine historical reasoning traces and balances the distribution of sparse but critical actions, leading to more coherent planning and better generalization in complex UI tasks. Our contributions include the publish of SOTA open-source UI agents, comprehensive data cleaning protocols and a novel self-evolving framework for improving navigation performance, which encourage further research and development in the community. Code is available at https://github.com/inclusionAI/UI-Venus.

cs.CV