Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Windows Agent Arena evaluates multi-modal OS agents; Navi achieves 19.5% success in Windows domain.
Key Findings
Methodology
The study uses the Windows Agent Arena framework, focusing on the Windows OS, with parallelized evaluation on Azure cloud. Tasks are designed based on Partially Observable Markov Decision Processes (POMDP), covering 154 multi-step tasks involving screen understanding and tool usage.
Key Results
- Navi achieves a 19.5% success rate in the Windows domain, significantly lower than the 74.5% human performance. It also performs well on the Mind2Web benchmark, demonstrating its multi-modal capabilities.
- Using Set-of-Marks prompting combined with system accessibility trees and pixel detectors, Navi excels in multiple tasks.
- Experiments show task parallelization reduces evaluation time to 20 minutes, significantly improving efficiency.
Significance
This study provides a scalable platform for evaluating multi-modal agents in real OS environments, bridging the gap between task complexity and simulation realism. It offers new research opportunities for agent development and data generation.
Technical Contribution
Windows Agent Arena offers a scalable evaluation environment supporting multi-modal agents in the Windows OS. The development of the Navi agent demonstrates task automation in complex OS environments.
Novelty
This is the first framework focused on evaluating multi-modal agents in the Windows OS, significantly extending the application scope of the OSWorld framework, providing a testing environment closer to human computing experiences.
Limitations
- Navi's success rate in the Windows domain is only 19.5%, indicating limitations in complex tasks.
- The current framework is primarily applicable to the Windows OS and may not be suitable for other operating systems.
Future Work
Future research directions include expanding task types, improving agent success rates, and exploring cross-OS multi-modal agent evaluation methods.
AI Executive Summary
Windows Agent Arena is a multi-modal agent evaluation platform focused on the Windows operating system. Existing evaluation benchmarks are often limited to specific modalities or domains, making it difficult to comprehensively assess agent performance in real environments. Windows Agent Arena, through parallelized evaluation, significantly reduces evaluation time and provides a scalable evaluation environment.
The Navi agent developed in the study achieves a 19.5% success rate in the Windows domain, significantly lower than the 74.5% human performance. Nonetheless, Navi performs well on other benchmarks such as Mind2Web, demonstrating its multi-modal capabilities. The study also demonstrates how to enhance agent task success rates using Set-of-Marks prompting combined with system accessibility trees and pixel detectors.
This study provides a scalable platform for evaluating multi-modal agents in real OS environments, bridging the gap between task complexity and simulation realism. Future research directions include expanding task types, improving agent success rates, and exploring cross-OS multi-modal agent evaluation methods.
Deep Analysis
Background
With the rapid development of large language models (LLMs), the potential for multi-modal agents in computer operating systems is immense. However, existing evaluation benchmarks are often limited to specific modalities or domains, making it difficult to comprehensively assess agent performance in real environments. Windows Agent Arena aims to address this issue by providing a multi-modal agent evaluation platform focused on the Windows OS.
Core Problem
Existing evaluation benchmarks have a significant gap between task complexity and simulation realism, making it difficult to comprehensively assess agent performance in real environments. Particularly in the Windows OS, there is a lack of a scalable evaluation environment to test multi-modal agent capabilities.
Innovation
Windows Agent Arena, through parallelized evaluation, significantly reduces evaluation time and provides a scalable evaluation environment. The development of the Navi agent demonstrates task automation in complex OS environments, bridging the gap between task complexity and simulation realism.
Methodology
- �� Use the Windows Agent Arena framework, focusing on the Windows OS.
- �� Tasks are designed based on Partially Observable Markov Decision Processes (POMDP).
- �� Evaluation is parallelized on Azure cloud, significantly improving efficiency.
- �� Use Set-of-Marks prompting combined with system accessibility trees and pixel detectors.
Experiments
The experimental design includes 154 multi-step tasks covering screen understanding and tool usage. Task parallelization reduces evaluation time to 20 minutes. Navi achieves a 19.5% success rate in the Windows domain and performs well on the Mind2Web benchmark.
Results
Navi achieves a 19.5% success rate in the Windows domain, significantly lower than the 74.5% human performance. It also performs well on the Mind2Web benchmark, demonstrating its multi-modal capabilities. Task parallelization significantly improves evaluation efficiency.
Applications
Windows Agent Arena can be used to evaluate multi-modal agent performance in the Windows OS, applicable to scenarios requiring complex task automation, such as software development and system management.
Limitations & Outlook
Navi's success rate in complex tasks is low, indicating limitations in task complexity and simulation realism. The current framework is primarily applicable to the Windows OS and may not be suitable for other operating systems.
Plain Language Accessible to non-experts
Imagine you're in a large kitchen with many different tools and ingredients. You need to follow a recipe to make a complex dish, but each step requires different tools and skills. Windows Agent Arena is like this kitchen, providing an environment where multi-modal agents act like chefs, completing various complex tasks in the Windows OS. The Navi agent is like a novice chef, able to complete some basic steps but needing improvement on complex dishes. Through continuous practice and improvement, Navi can gradually handle more complex tasks.
ELI14 Explained like you're 14
Imagine you're playing a super complex computer game that requires you to switch between different windows and complete various tasks. Windows Agent Arena is like the stage for this game, and Navi is your game character. Although Navi can't yet perform as well as human players, it's learning how to survive and thrive in this complex game world. In the future, Navi might become smarter and able to complete more tasks, just like you level up and progress in a game.
Glossary
Multi-modal Agent
An intelligent agent capable of processing multiple input modalities, such as text and images.
Used to evaluate agent performance in the Windows OS.
Windows Agent Arena
A multi-modal agent evaluation platform focused on the Windows OS.
Provides a scalable evaluation environment.
Navi
A multi-modal agent developed in the study, focused on the Windows OS.
Achieves a 19.5% success rate in the Windows domain.
Set-of-Marks
A prompting method combining system accessibility trees and pixel detectors.
Used to enhance agent task success rates.
Azure Cloud
A cloud computing service platform provided by Microsoft.
Used for parallelized evaluation, significantly improving efficiency.
Open Questions Unanswered questions from this research
- 1 How to improve Navi's success rate in complex tasks? Existing methods have limitations in task complexity and simulation realism.
- 2 How to extend Windows Agent Arena to support other operating systems?
- 3 How to achieve more efficient task automation in multi-modal agents?
Applications
Immediate Applications
Software Development
Evaluate multi-modal agent performance in software development using Windows Agent Arena to improve development efficiency.
Long-term Vision
System Management
Utilize multi-modal agents to automate complex system management tasks, reducing labor costs and improving efficiency.
Abstract
Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena. Webpage: https://microsoft.github.io/WindowsAgentArena Code: https://github.com/microsoft/WindowsAgentArena