How Benchmarks Mis-Score Computer-Use Agents
The study reveals benchmark scoring errors for computer-use agents, proposing a four-stage reliability framework with 15.3% incorrect failure verdicts.
Key Findings
Methodology
The study employs a four-stage reliability framework: task construction, trajectory observation, scoring, and reporting. It audits 150 failure-scored trajectories, revealing issues in task construction contamination, evaluator fragility, and diagnostic opacity. Uses MAST taxonomy for fault diagnosis, analyzing evaluator false negatives and broken tasks.
Key Results
- Result 1: Of the 150 audited trajectories, 15.3% of failure verdicts are wrong, with 10.7% evaluator false negatives and 4.7% broken tasks.
- Result 2: Feedback blindness and planning errors dominate execution and grounding errors, which cannot be explained by a single success rate.
- Result 3: The audit proposes stage-specific design rules for CUA evaluation.
Significance
This study highlights the scoring errors in current CUA benchmarks and proposes a reliability framework to improve evaluation accuracy and transparency. It is significant for academia and industry, especially in long-horizon tasks and multi-application workflows, addressing gaps in existing evaluation methods.
Technical Contribution
Technical contributions include a new four-stage reliability framework, systematically analyzing the sources of scoring errors, and providing specific design rules to improve CUA evaluation. Compared to existing methods, this framework focuses more on task validity and evaluator reliability.
Novelty
This is the first study to systematically analyze CUA benchmark scoring errors, proposing a four-stage reliability framework that addresses deficiencies in existing evaluation methods for long-horizon tasks.
Limitations
- Limitation 1: The study is limited to five benchmarks and may not cover all CUA application scenarios.
- Limitation 2: Identification of evaluator false negatives and broken tasks depends on the completeness of trajectories.
Future Work
Future research can expand to more CUA application scenarios to further validate the framework's applicability. Developing smarter evaluation tools to reduce false negatives and task breakages is also an important direction.
AI Executive Summary
Computer-use agents (CUA) are increasingly used to browse the web and operate desktop software, yet their benchmark scores often rely on fragile scripted evaluators, leading to incorrect failure verdicts. This study audits 150 failure-scored trajectories, finding 15.3% of failure verdicts incorrect. The proposed four-stage reliability framework covers task construction, trajectory observation, scoring, and reporting, aiming to improve evaluation accuracy and transparency.
The study reveals critical issues in current CUA evaluations, such as task construction contamination, evaluator fragility, and diagnostic opacity. Using the MAST taxonomy for fault diagnosis, it finds that feedback blindness and planning errors dominate execution and grounding errors.
The significance of this study lies in providing a more reliable framework for CUA evaluation, enhancing accuracy in long-horizon tasks and multi-application workflows. This framework not only addresses gaps in existing methods but also guides future CUA evaluation tool development.
Deep Analysis
Background
With the widespread use of computer-use agents (CUA), evaluating their performance in web browsing and desktop software operation has become crucial. Traditional evaluation methods rely on scripted evaluators, which are prone to misjudgments, especially in complex multi-application workflows. This study aims to uncover these evaluation shortcomings and propose improvements.
Core Problem
CUA evaluations often rely on fragile scripted evaluators, leading to incorrect failure verdicts. The core problem is how to improve evaluation accuracy and transparency to better reflect CUA's actual capabilities.
Innovation
The study proposes a four-stage reliability framework covering task construction, trajectory observation, scoring, and reporting. Each stage has unique challenges and solutions, particularly in task construction contamination, evaluator fragility, and diagnostic opacity.
Methodology
- �� Task Construction: Ensure task validity and independence.
- �� Trajectory Observation: Record complete trajectories to support diagnosis.
- �� Scoring: Recognize valid outcomes and verify each constraint.
- �� Reporting: Provide detailed failure diagnosis and process-stage analysis.
Experiments
The study audits 150 failure-scored trajectories from five benchmarks. Using the MAST taxonomy for fault diagnosis, it analyzes evaluator false negatives and broken tasks. The experimental results reveal shortcomings in current evaluation methods and propose improvements.
Results
Audit results show that 15.3% of failure verdicts are incorrect, with 10.7% evaluator false negatives and 4.7% broken tasks. Feedback blindness and planning errors dominate execution and grounding errors, which cannot be explained by a single success rate.
Applications
The study's findings can improve CUA evaluation methods, especially in long-horizon tasks and multi-application workflows. By enhancing evaluation accuracy and transparency, it better reflects CUA's actual capabilities.
Limitations & Outlook
The study is limited to five benchmarks and may not cover all CUA application scenarios. Additionally, the identification of evaluator false negatives and broken tasks depends on the completeness of trajectories, requiring further validation of the framework's applicability.
Plain Language Accessible to non-experts
Imagine you're in a kitchen preparing a big meal. You need to follow a recipe to prepare various ingredients, ensuring each step is correct. CUAs are like chefs in the kitchen, and benchmarks are like the recipe. Traditional evaluation methods are like an unreliable assistant who might miss key steps or misjudge results. The study proposes a new method, like equipping the chef with a more reliable assistant, ensuring each step is accurately executed.
ELI14 Explained like you're 14
Imagine you're playing a complex video game where you need to complete multiple tasks to level up. CUAs are like the characters in the game, and benchmarks are like the game's rules. Sometimes, the rules mess up, preventing you from leveling up. This study is like a big update to the game rules, making them more accurate so you can complete tasks better. Isn't that cool?
Glossary
Computer-Use Agent
A software agent capable of automatically performing tasks on a computer, such as web browsing and software operation.
Used in the study to evaluate its performance in complex tasks.
Benchmark
A standardized test used to evaluate the performance of a system or software.
Used in the study to assess CUA performance.
False Negative
An evaluation that incorrectly judges a correct result as wrong.
Analyzed in the study for its impact on scoring.
Trajectory
The complete process record of a CUA performing a task, including each step's actions and results.
Used in the study to analyze the sources of scoring errors.
Reliability Framework
A structured approach to improving evaluation accuracy and transparency.
The four-stage reliability framework proposed in the study.
Open Questions Unanswered questions from this research
- 1 How to validate the framework's applicability in more CUA application scenarios?
- 2 How to develop smarter evaluation tools to reduce false negatives?
Applications
Immediate Applications
CUA Evaluation Improvement
By improving evaluation accuracy, it helps developers better understand CUA's actual capabilities.
Long-term Vision
Intelligent Evaluation Tools
Develop smarter evaluation tools to reduce false negatives and task breakages, enhancing CUA evaluation reliability.
Abstract
Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong: 10.7\% are evaluator false negatives and 4.7\% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.