ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
ShortcutsBench provides a large-scale benchmark to evaluate API agents in complex tasks.
Key Findings
Methodology
ShortcutsBench integrates real APIs from Apple, user queries, and human-annotated high-quality action sequences to provide a comprehensive evaluation framework. It covers API selection, parameter filling, and the ability to request input from the system or user.
Key Results
- Existing API agents show significant limitations in handling complex queries, with lower than expected accuracy in API selection.
- Parameter filling, especially extracting necessary parameters from queries, presents a major challenge.
- Agents lack sufficient awareness in requesting necessary input, leading to low task completion rates.
Significance
This research provides new insights into the deficiencies of API agents in handling complex queries, revealing the limitations of current methods in real-world applications. ShortcutsBench allows researchers to better understand and improve API agent performance.
Technical Contribution
ShortcutsBench is the first large-scale real API benchmark offering rich APIs and task types, supporting comprehensive evaluation of API agents. It introduces parameter filling and input request evaluation, filling gaps in existing benchmarks.
Novelty
ShortcutsBench is the first to combine real APIs and user demands, providing a comprehensive evaluation framework that surpasses previous benchmarks focused only on API selection.
Limitations
- Current API agents perform poorly in multi-step tasks, especially in parameter filling and input requests.
- The benchmark is mainly based on Apple's APIs, which may limit its applicability to other platforms.
Future Work
Future research could expand to APIs from more platforms, explore smarter parameter filling methods, and enhance agents' ability to request input.
AI Executive Summary
Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have garnered significant interest in academia and industry. However, the ability of these agents to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unclear. ShortcutsBench integrates real APIs from Apple, user queries, and human-annotated high-quality action sequences to provide a comprehensive evaluation framework. Experimental results reveal significant limitations of existing API agents in handling complex queries, particularly in API selection, parameter filling, and input requests. This research provides new insights into the deficiencies of API agents in real-world applications and points to future research directions.
ShortcutsBench is designed based on data extracted from Apple Shortcuts, offering rich APIs and task types, supporting comprehensive evaluation of API agents. Extensive evaluations of 10 leading open-source and closed-source LLMs reveal the deficiencies of existing API agents in handling complex queries, particularly in parameter filling and input requests. Results indicate that open-source LLMs now match closed-source ones on simpler tasks but still lag behind on complex ones.
Future research could expand to APIs from more platforms, explore smarter parameter filling methods, and enhance agents' ability to request input. The open-source nature of ShortcutsBench provides researchers with rich resources, advancing the development of API agents in real-world applications.
Deep Analysis
Background
In recent years, the integration of large language models (LLMs) with application programming interfaces (APIs) has gained significant interest in academia and industry. By combining LLMs with APIs, agents can access real-time information, reduce hallucinations, and plan and complete complex tasks requiring multi-step actions. However, existing benchmarks struggle to evaluate agents' ability to handle complex tasks effectively.
Core Problem
The ability of existing API agents to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unclear. Particularly, agents show significant limitations in API selection, parameter filling, and input requests.
Innovation
ShortcutsBench integrates real APIs from Apple, user queries, and human-annotated high-quality action sequences to provide a comprehensive evaluation framework. It is the first to combine real APIs and user demands, surpassing previous benchmarks focused only on API selection.
Methodology
- �� Collect APIs and user queries from Apple Shortcuts
- �� Provide human-annotated high-quality action sequences
- �� Evaluate API selection, parameter filling, and input request abilities
- �� Conduct extensive evaluations using 5 open-source and 5 closed-source LLMs
Experiments
The experimental design includes extensive evaluations of 10 leading open-source and closed-source LLMs, covering API selection, parameter filling, and input request abilities. Benchmarks used include MetaTool and ToolBench.
Results
Experimental results reveal significant limitations of existing API agents in handling complex queries, particularly in API selection, parameter filling, and input requests. Open-source LLMs now match closed-source ones on simpler tasks but still lag behind on complex ones.
Applications
ShortcutsBench provides a comprehensive framework for evaluating and improving API agents' performance in handling complex tasks. It offers researchers rich resources, advancing the development of API agents in real-world applications.
Limitations & Outlook
Current API agents perform poorly in multi-step tasks, especially in parameter filling and input requests. The benchmark is mainly based on Apple's APIs, which may limit its applicability to other platforms.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. An API agent is like a kitchen assistant that helps you find recipes, prepare ingredients, and follow cooking steps. ShortcutsBench is like a big cookbook with various complex recipes, helping evaluate these assistants' performance in handling complex tasks. However, current assistants perform poorly in multi-step recipes, especially in selecting the right ingredients and steps. ShortcutsBench provides rich recipes and detailed steps to help us better understand and improve these assistants' performance.
ELI14 Explained like you're 14
Imagine you're playing a complex game, and you have an assistant to help you complete tasks. This assistant can help you find information, choose tools, and complete tasks. ShortcutsBench is like a game guide with various complex tasks, helping evaluate these assistants' performance in handling complex tasks. However, current assistants perform poorly in multi-step tasks, especially in selecting the right tools and steps. ShortcutsBench provides rich tasks and detailed steps to help us better understand and improve these assistants' performance.
Glossary
API (Application Programming Interface)
APIs are interfaces for communication between applications, allowing different software systems to interact.
In this paper, APIs are used to evaluate agents' ability to handle complex tasks.
LLM (Large Language Model)
LLMs are models trained on large text datasets, capable of generating and understanding natural language.
In this paper, LLMs are combined with APIs to create intelligent agents.
ShortcutsBench
ShortcutsBench is a large-scale benchmark for evaluating API agents in complex tasks.
The paper introduces ShortcutsBench as an evaluation framework.
Parameter Filling
Parameter filling refers to filling in necessary parameters in API calls to complete tasks.
The paper evaluates agents' performance in parameter filling tasks.
Input Request
Input request refers to agents requesting additional information from the system or user when needed.
The paper evaluates agents' ability to request input.
Open Questions Unanswered questions from this research
- 1 Current API agents perform poorly in multi-step tasks, especially in parameter filling and input requests.
- 2 The benchmark is mainly based on Apple's APIs, which may limit its applicability to other platforms.
Applications
Immediate Applications
API Agent Evaluation
ShortcutsBench can be used to evaluate and improve API agents' performance in handling complex tasks.
Long-term Vision
Cross-Platform Applications
Future research could expand to APIs from more platforms, exploring smarter parameter filling methods.
Abstract
Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. In this paper, we introduce \textsc{ShortcutsBench}, a large-scale benchmark for the comprehensive evaluation of API-based agents in solving real-world complex tasks. \textsc{ShortcutsBench} includes a wealth of real APIs from Apple Inc., refined user queries, human-annotated high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing benchmarks~/~datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with $5$ leading open-source (size $\geq$ 57B) and $5$ closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-4o-mini) with varying intelligence level reveals significant limitations of existing API-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting necessary input from the system and the user. These findings highlight the great challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at \url{https://github.com/EachSheep/ShortcutsBench}.