API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

Recent advances in LLMs like GPT-3, Codex, and GPT-4 have revolutionized NLP, enabling tasks from conversation to code generation. Despite these breakthroughs, models are limited by static training data, which quickly becomes outdated. External tools, such as search engines and APIs, have been integrated to supplement knowledge, exemplified by systems like WebGPT and ReAct. However, existing evaluation frameworks lack comprehensive, real-world scenarios that test models' ability to plan, retrieve, and call diverse APIs across domains. This gap hampers progress in tool-augmented AI, restricting deployment in dynamic, multi-task environments.

The core challenge is the absence of a unified, realistic benchmark to evaluate models' tool utilization capabilities systematically. Current models show inconsistent performance in API calling, often failing in multi-step, multi-API tasks due to poor planning, retrieval errors, and coordination issues. This limits their practical application in complex workflows like automated customer support, personalized assistants, and real-time data retrieval. Developing a benchmark that reflects real-world complexities and provides actionable insights is crucial for advancing the field.

This paper introduces several innovations: 1) A comprehensive, executable evaluation system with 73 APIs covering multiple domains, enabling real-time assessment of API call, retrieval, and planning abilities. 2) A multi-agent data generation framework that automates the creation of high-quality training data, reducing annotation costs by 98%. 3) Training of Lynx, a model based on Alpaca-7B, with improved tool utilization capabilities. 4) A detailed analysis of model performance across different abilities and scenarios, highlighting bottlenecks and guiding future research. These innovations collectively push the frontier of tool-augmented LLM research, offering scalable evaluation and training solutions.

• �� Constructed a real API system with 73 APIs, covering weather, database, AI services, etc. • Manually implemented APIs, establishing databases and static information for reproducibility. • Annotated 314 dialogues with 753 API calls, covering call, retrieval, and planning tasks. • Developed a multi-agent system: • Agent 1 generates domains (health, finance, etc.). • Agent 2 creates APIs based on domain info. • Agent 3 selects APIs and abilities, generating user queries. • Agent 4 simulates API calls and responses, ensuring correctness. • Agent 5 tests data quality, discarding invalid samples. • Fine-tuned Lynx on 1888 dialogues, evaluated on a separate test set. • Compared models like GPT-3, GPT-3.5, GPT-4, and Lynx across various tasks, measuring accuracy and response quality.

The evaluation involved testing GPT-3, GPT-3.5, GPT-4, ChatGLM-6B, Alpaca-7B, and Lynx on the API-Bank benchmark. Metrics included API call accuracy, retrieval success, and response ROUGE-L scores. Experiments covered three task types: single API call, multiple API calls, and combined planning-retrieval-calling. Hyperparameters: batch size 256, learning rate 2e-5, trained Lynx for three epochs. Results showed GPT-3.5 and GPT-4 outperform smaller models, with GPT-4 achieving over 70% in multi-API tasks. Ablation studies confirmed the importance of training data quality and multi-agent generation. Error analysis identified bottlenecks in API selection, parameter passing, and multi-API orchestration, guiding future improvements.

GPT-3.5's API call accuracy reached 60%, a substantial improvement over GPT-3's near-zero performance. Lynx, trained on the new dataset, improved by 26 points over Alpaca-7B, approaching GPT-3.5 levels. GPT-4 achieved over 70% accuracy in multi-API tasks, demonstrating strong planning and coordination. Error analysis revealed that models struggle with API selection, especially in large API pools, and with multi-step reasoning. These results validate the effectiveness of the benchmark and training approach, highlighting key areas for future research to close remaining gaps.

The benchmark enables systematic evaluation of models for real-world applications like intelligent virtual assistants, automated customer service, and dynamic data retrieval systems. Models trained with API-Bank can better handle complex multi-step workflows, improving user experience and operational efficiency. The framework also supports continuous improvement through iterative training and feedback, fostering AI systems that adapt to evolving needs and environments.

Models still face challenges in multi-API coordination, especially in complex, multi-step tasks requiring precise parameter passing. The training data, although large, may lack sufficient coverage of niche domains, limiting performance in specialized fields. The static evaluation environment does not fully simulate real-time, dynamic scenarios such as live web data or user interactions. Future work should focus on adaptive learning, environment modeling, and expanding API diversity to address these issues.