OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models

In recent years, Vision-Language Models (VLMs) have demonstrated remarkable performance in general vision-language tasks and show potential in understanding, navigating, and manipulating graphical user interfaces (GUIs). Computer-using agents (CUAs) interact with GUI environments (e.g., desktop, mobile, web) to autonomously complete user-specified tasks, laying a solid foundation for digital task automation. However, current CUAs still face issues such as operation failure, inefficient error state recovery, mistimed task termination, and inaccurate element localization.

CUAs tend to accumulate errors over long workflows, leading to unintended consequences. The effectiveness of critic models is hindered by the lack of diverse, high-quality GUI feedback data and public critic benchmarks for step-level evaluation in computer use. To address these issues, the OS-Oracle framework provides a scalable data pipeline and a two-stage training method to enhance critic model performance.

The core innovations of OS-Oracle lie in its automated data pipeline and two-stage training strategy. The automated data pipeline generates high-quality critic datasets, filling the gap in multi-platform data. The two-stage training strategy combines supervised fine-tuning and Consistency-Preserving Group Relative Policy Optimization (CP-GRPO), significantly enhancing the accuracy and consistency of model judgments. Additionally, the OS-Critic Bench serves as a cross-platform evaluation benchmark, providing comprehensive evaluation standards.

• �� Automated Data Pipeline: Extracts positive samples from existing trajectory datasets and generates negative samples for specific error types using rules.
• �� Two-Stage Training Strategy: Begins with supervised fine-tuning to establish core discrimination and reasoning skills, followed by Consistency-Preserving Group Relative Policy Optimization (CP-GRPO) to improve discriminability and reasoning-judgment agreement.
• �� OS-Critic Bench: Samples trajectories from existing computer-using benchmarks and engages human experts to assign binary labels, ensuring high-quality datasets.

The experimental design involves using the Qwen2.5-VL-7B-Instruct model for two-stage training on the OS-Oracle dataset. It begins with supervised fine-tuning, followed by further optimization in the CP-GRPO phase. Evaluation is based on the OS-Critic Bench, covering mobile, web, and desktop platforms. Results show that OS-Oracle-7B achieves performance improvements in multiple online benchmarks, such as increasing OSWorld from 29.2% to 31.0%.

OS-Oracle-7B excels on the OS-Critic Bench, particularly surpassing proprietary models in the mobile domain, demonstrating cross-platform capabilities across mobile, desktop, and web platforms. As a pre-critic, OS-Oracle-7B enhances the performance of UI-TARS-1.5-7B in OSWorld and AndroidWorld environments. Experimental results show performance improvements in multiple online benchmarks, such as increasing OSWorld from 29.2% to 31.0%.

The OS-Oracle framework has broad application prospects in multi-platform GUI critic models. It can improve the performance of existing GUI agents, particularly on mobile, desktop, and web platforms. Additionally, OS-Oracle can serve as a foundation for critic models, further advancing the development of critic models in academia and industry.

Although OS-Oracle performs well in multi-platform critic models, it may underperform in handling extremely complex GUI tasks, especially in scenarios not covered by the dataset. Additionally, the current critic model may have limited response speed in real-time applications. Future research can focus on expanding the diversity and scale of datasets and improving the response speed and accuracy of critic models in real-time applications.