AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
AgentStore improves OSWorld benchmark performance to 23.85% using MetaAgent and AgentToken strategy.
Key Findings
Methodology
AgentStore is a scalable platform for dynamically integrating heterogeneous agents to automate computer tasks. Its core comprises MetaAgent for agent management and AgentToken strategy for selecting suitable agents via learnable embeddings.
Key Results
- On the OSWorld benchmark, AgentStore's success rate increased from 11.21% to 23.85%, significantly outperforming previous systems.
- On mobile platforms, AgentStore achieved a success rate of 57.8%, demonstrating cross-platform adaptability.
- Experiments show that the AgentToken strategy excels in dynamically managing a large number of agents.
Significance
AgentStore addresses the limitations of existing systems in handling open-ended tasks by integrating diverse agents, advancing the development of specialized generalist computer assistants.
Technical Contribution
AgentStore introduces MetaAgent and AgentToken strategies, providing a novel method for dynamically managing a large number of heterogeneous agents, surpassing the capabilities of existing single-agent systems.
Novelty
AgentStore is the first to introduce the App Store concept into the digital agent domain, achieving task automation through dynamic integration of heterogeneous agents.
Limitations
- In some complex tasks, agent collaboration still needs optimization, which may affect efficiency.
- Integration of new agents requires a learning process.
Future Work
Future work includes optimizing agent collaboration mechanisms, expanding AgentStore's application in more operating systems, and enhancing system adaptability.
AI Executive Summary
AgentStore addresses the limitations of existing systems in handling open-ended tasks by integrating diverse agents. Its core comprises MetaAgent for agent management and AgentToken strategy for selecting suitable agents via learnable embeddings. Experiments show that AgentStore's success rate on the OSWorld benchmark increased from 11.21% to 23.85%, and on mobile platforms, it achieved 57.8%. This demonstrates its cross-platform adaptability and superior performance in handling complex tasks. Future work includes optimizing agent collaboration mechanisms, expanding AgentStore's application in more operating systems, and enhancing system adaptability.
Deep Analysis
Background
As operating systems and applications evolve, users face steeper learning curves. Digital agents offer potential for automating complex computer tasks, but existing methods perform poorly on open-ended tasks.
Core Problem
Existing agent systems lack generalization and specialization capabilities for handling open-ended tasks, especially in real-world environments. Effectively integrating diverse agents to enhance system capabilities is a key challenge.
Innovation
AgentStore introduces MetaAgent and AgentToken strategies for dynamic integration of heterogeneous agents. MetaAgent manages agents, while AgentToken selects suitable agents via learnable embeddings.
Methodology
- �� Develop AgentStore prototype and establish agent integration protocol. • Introduce MetaAgent using AgentToken strategy for selecting suitable agents. • Validate through experiments on OSWorld benchmark.
Experiments
On the OSWorld benchmark, AgentStore achieved a success rate of 23.85%, significantly outperforming previous systems. Experiments used InternVL2-8B as the base model, with multiple iterative training.
Results
AgentStore's success rate on the OSWorld benchmark increased from 11.21% to 23.85%, and on mobile platforms, it achieved 57.8%. This demonstrates its cross-platform adaptability and superior performance in handling complex tasks.
Applications
AgentStore can be used for automating operating system tasks, particularly suitable for complex task scenarios requiring multi-agent collaboration.
Limitations & Outlook
In some complex tasks, agent collaboration still needs optimization, which may affect efficiency. Integration of new agents requires a learning process.
Plain Language Accessible to non-experts
Imagine a large supermarket with various specialized shops, each with its own expertise. AgentStore is like this supermarket, integrating different shops to meet diverse customer needs. MetaAgent is the supermarket manager, coordinating the work of each shop, while AgentToken is the shop's identifier, helping the manager quickly find the right shop to complete tasks.
ELI14 Explained like you're 14
Imagine you're playing a game with many characters, each with their own skills. AgentStore is like a collection of characters you can choose from to complete tasks. MetaAgent is the game commander, arranging the work of the characters, while AgentToken is like the character's name tag, helping the commander quickly find the right character.
Glossary
MetaAgent
The core component responsible for managing and coordinating all agents in AgentStore.
Used to select suitable agents for task completion.
AgentToken
A learnable embedding representing each agent in AgentStore.
Helps MetaAgent select suitable agents.
OSWorld
A real-world environment benchmark for evaluating computer agents, comprising 369 tasks.
Used to test AgentStore's capability range.
App Store
A platform offering diverse applications; AgentStore borrows its concept.
Used to analogize AgentStore's capability expansion.
In-Context Learning
A method for learning new tasks through context information without retraining the model.
Used in AgentStore for agent selection.
Open Questions Unanswered questions from this research
- 1 How to further optimize agent collaboration mechanisms to enhance efficiency in complex tasks?
- 2 What are the challenges of applying AgentStore in more operating systems?
Applications
Immediate Applications
Operating System Automation
AgentStore can be used to automate complex operating system tasks, enhancing user efficiency.
Long-term Vision
Cross-Platform Intelligent Assistant
AgentStore has the potential to develop into a cross-platform intelligent assistant supporting multiple operating systems.
Abstract
Digital agents capable of automating complex computer tasks have attracted considerable attention due to their immense potential to enhance human-computer interaction. However, existing agent methods exhibit deficiencies in their generalization and specialization capabilities, especially in handling open-ended computer tasks in real-world environments. Inspired by the rich functionality of the App store, we present AgentStore, a scalable platform designed to dynamically integrate heterogeneous agents for automating computer tasks. AgentStore empowers users to integrate third-party agents, allowing the system to continuously enrich its capabilities and adapt to rapidly evolving operating systems. Additionally, we propose a novel core \textbf{MetaAgent} with the \textbf{AgentToken} strategy to efficiently manage diverse agents and utilize their specialized and generalist abilities for both domain-specific and system-wide tasks. Extensive experiments on three challenging benchmarks demonstrate that AgentStore surpasses the limitations of previous systems with narrow capabilities, particularly achieving a significant improvement from 11.21\% to 23.85\% on the OSWorld benchmark, more than doubling the previous results. Comprehensive quantitative and qualitative results further demonstrate AgentStore's ability to enhance agent systems in both generalization and specialization, underscoring its potential for developing the specialized generalist computer assistant. All our codes will be made publicly available in https://chengyou-jia.github.io/AgentStore-Home.