ToolACE: Winning the Points of LLM Function Calling
ToolACE employs self-evolution synthesis of 26,507 APIs, enabling high-performance function calling with only 8B parameters, rivaling GPT-4.
Key Findings
Methodology
ToolACE integrates a self-evolution synthesis (TSS) framework, building hierarchical API trees from pretraining data, and employs multi-agent dialogue generation guided by complexity evaluators. It uses a dual-layer verification system combining rule-based and model-based checks to ensure data quality. The training employs LoRA fine-tuning on an 8B parameter model, achieving state-of-the-art results on BFCL and API-Bank benchmarks, with accuracy surpassing comparable open-source models and close to GPT-4.
Key Results
- On BFCL, ToolACE-8B achieved 89.27% accuracy in AST tasks and maintained high multi-turn and dependency call performance, outperforming xLAM-8x22b-r and Gorilla-OpenFunctions-v2. In API-Bank, accuracy reached 75.94%, significantly surpassing similar-sized open models. The dual verification reduced hallucination to 14.37% and improved relevance scores to 85.37%, ensuring high data integrity.
- Synthetic data diversity and complexity contributed to enhanced zero-shot generalization, with the model ranking third overall in BFCL, demonstrating comparable performance to GPT-4 series in function calling tasks.
- The multi-agent dialogue generation, guided by model-based complexity assessment, produced varied, multi-turn, dependent, and non-tool dialogues, effectively covering real-world scenarios and reducing data bias.
Significance
This work addresses critical bottlenecks in scaling function calling capabilities of LLMs by automating diverse, accurate, and complex data synthesis. It significantly advances the ability of models to handle multi-tool, multi-turn interactions, essential for real-world AI applications like automation, virtual assistants, and intelligent workflows. The approach reduces reliance on costly manual data annotation, enabling scalable deployment in industry, and sets a new benchmark for synthetic data quality and diversity in tool-augmented LLMs.
Technical Contribution
The paper introduces an innovative API self-evolution mechanism, multi-agent dialogue generation with complexity regulation, and a robust dual-layer validation system. These components collectively enhance data diversity, complexity, and correctness, enabling training of smaller models that rival large-scale counterparts. The framework offers a scalable, automated pipeline for generating high-fidelity training data, pushing forward the frontier of tool-augmented LLM capabilities.
Novelty
This is the first comprehensive system integrating API self-evolution, multi-agent dialogue synthesis, and multi-level validation to generate diverse, complex, and accurate function-calling data. Unlike prior works relying on static API datasets, ToolACE dynamically evolves APIs and dialogues, vastly expanding the scope and depth of training data, which directly improves model generalization and robustness.
Limitations
- API self-evolution heavily depends on pretraining data richness, potentially limiting adaptation to emerging or niche domains. The synthetic dialogues, while diverse, may not fully capture real user behaviors, requiring further fine-tuning with real data.
- The validation system, though effective, might still miss rare errors in highly complex scenarios, necessitating more advanced verification techniques. Computational costs of multi-agent generation and validation also pose scalability challenges.
- Model performance in extremely low-resource or highly specialized fields remains uncertain, calling for future research on domain-specific API synthesis and validation.
Future Work
Future directions include integrating real-world API updates for continuous evolution, expanding multi-modal capabilities, and applying reinforcement learning to optimize dialogue complexity. Enhancing validation with automated human-in-the-loop systems and extending to multilingual, multi-modal tasks will further broaden the applicability of ToolACE in industry and research.
AI Executive Summary
The rapid development of large language models (LLMs) has unlocked unprecedented capabilities in natural language understanding and generation. However, extending these models to effectively invoke external tools and APIs remains a significant challenge, primarily due to the scarcity of high-quality, diverse training data. Existing datasets are often limited in scope, static, or reliant on manual annotation, constraining models’ ability to generalize across complex, multi-turn, multi-tool scenarios.
To address this, ToolACE introduces a novel, fully automated data synthesis pipeline that leverages self-evolution mechanisms to generate a vast, diverse API pool of over 26,500 APIs spanning multiple domains. This API pool is dynamically expanded through a speciation-adaptation-evolution process, ensuring broad coverage and detailed functionalities. Concurrently, multi-agent dialogue generation simulates realistic user-assistant-tool interactions, guided by a model-based complexity evaluator that ensures data complexity matches the model’s learning stage.
A key innovation is the dual-layer verification system, which combines rule-based checks for syntactic correctness with model-based content validation, significantly reducing errors and hallucinations. The entire framework enables training of a relatively small 8B parameter model, which achieves performance comparable to GPT-4 on benchmarks like BFCL and API-Bank, surpassing many open-source counterparts.
This work marks a substantial step forward in automating high-quality data generation for function calling, reducing reliance on manual annotation, and enhancing model robustness in real-world applications. It opens avenues for scalable, multi-domain AI systems capable of complex tool usage, with broad implications for automation, virtual assistants, and intelligent workflows. Future enhancements will focus on integrating real API updates, expanding multi-modal capabilities, and refining validation techniques to further push the boundaries of AI tool integration.
Deep Analysis
Background
近年来，随着GPT-3、T5等模型的广泛应用，LLMs在自然语言处理中的地位不断提升。早期研究如GPT-3、T5主要关注模型能力的提升，随后逐步转向模型与外部工具结合，如Toolformer、PALM等，旨在增强模型的任务执行能力。然而，真实API的快速变化和多样性带来了数据采集和标注的巨大挑战。现有的合成数据多为静态样本，缺乏多轮、多工具、多复杂参数的场景，限制了模型的泛化能力。近年来，合成数据的应用逐渐兴起，但缺乏系统性和多样性，难以满足实际应用需求。因此，自动化生成高质量、多样化API调用数据成为研究热点，亟需突破技术瓶颈。
Core Problem
尽管大模型在自然语言理解和生成方面取得了显著突破，但其在复杂工具调用场景中的表现仍受限。现有数据多为静态、单一API调用样本，难以覆盖多轮交互、依赖调用和复杂参数结构，限制模型在真实场景中的适应性。真实API的频繁更新和多样性增加了数据采集难度，人工标注成本高昂，难以实现大规模覆盖。如何自动生成丰富、多样、准确且具有挑战性的训练数据，提升模型在多任务、多场景中的泛化能力，成为核心难题。同时，确保数据的质量和多样性，避免偏差和错误，也是实现高性能模型的关键。
Innovation
本研究提出ToolACE框架，核心创新包括：1）API自我演化机制，通过层级API树自动提取多领域API，丰富数据池；2）多智能体对话生成，模拟用户、助手和工具的交互，生成多样化复杂对话场景；3）模型引导的复杂度调节，根据模型的损失指标动态调整对话难度，确保数据既具有挑战性又符合模型能力；4）双层验证体系，结合规则和模型检测，确保数据的正确性和一致性。这些创新突破了传统依赖静态API和手工标注的局限，为大模型的泛化能力提供了坚实基础。
Methodology
- �� API自我演化：利用预训练数据中的API文档，递归构建API层级树，提取不同领域的API定义，加入多样参数和约束，生成新API。
- �� 多智能体对话：模拟用户、助手、工具三方交互，生成单轮、多轮、依赖调用等多样场景，采用多次一致性筛选确保质量。
- �� 复杂度调节：用模型作为评估器，基于损失指标动态调节对话复杂度，确保数据既不过于简单也不过于复杂。
- �� 双层验证：规则层确保格式和结构正确，模型层检测内容合理性和一致性，结合人工验证提升可靠性。
Experiments
采用BFCL和API-Bank两个基准，训练参数为8B，使用LoRA微调。对比GPT-4、Gorilla-OpenFunctions等模型，评估函数调用准确率、Hallucination和Irrelevance指标。通过消融实验验证API多样性、复杂度调节和验证体系的贡献，分析模型在多轮、多工具场景中的表现。实验显示，ToolACE-8B在准确率和鲁棒性方面优于多数开源模型，验证了合成数据的有效性。
Results
模型在BFCL中AST类别准确率达89.27%，多轮和依赖调用场景表现优越，显著优于对比模型。在API-Bank中，准确率达75.94%，优于xLAM-8x22b-r和Gorilla-OpenFunctions-v2。验证体系显著降低Hallucination（14.37%）和Irrelevance（83.81%），确保数据质量。多样化API和复杂对话提升模型泛化能力，验证了合成数据的实用性和有效性。
Applications
该技术可广泛应用于智能助理、自动化工作流、金融分析等场景，支持多轮、多工具交互，提升自动化水平。未来结合真实API和多模态数据，将推动工业界智能系统的自主学习和适应能力，降低数据采集成本，增强模型的实用性。
Limitations & Outlook
当前API演化依赖预训练数据，可能在新兴或低资源领域表现不足。合成对话虽丰富，但与真实场景仍有差距，需结合实际数据微调。验证体系在极端复杂场景下可能漏检，未来需引入更强的自动化验证机制。模型训练成本和复杂度仍需优化，以实现更广泛的应用。
Plain Language Accessible to non-experts
想象你在一家大型厨房里做菜。每次做菜都需要不同的调料和工具，厨房里的调料罐和工具箱就像API一样。以前，厨师只知道少量调料，遇到新菜就得花时间找配料。现在，厨师有了一个智能助手，能自己学习新调料的配方，还能根据不同菜式自动搭配调料。这个助手会不断学习新菜谱，帮厨师准备各种复杂菜肴，并确保每次用料正确，避免出错。这样一来，厨房的菜肴变得丰富多样，厨师也能更快做出美味佳肴。ToolACE就像这个智能助手，能自己学习API，生成丰富的对话，帮助模型更聪明地调用工具，做出更复杂的任务。
ELI14 Explained like you're 14
想象你在学校的科学实验室里，老师让你用各种工具做实验。有时候，你需要用显微镜观察细胞，有时候用化学试剂反应。以前，你只知道一些基础工具，遇到新实验就得问老师或者查资料。现在，假如你有一个超级聪明的机器人助手，它可以自己学习各种新工具的用法，还能帮你设计复杂的实验方案。它会不断学习新工具的说明书，帮你模拟实验过程，确保每一步都正确。这样一来，你可以用它做很多复杂的实验，不用担心出错，也不用花太多时间查资料。ToolACE就像这个机器人助手，能自己学会新工具，帮模型更聪明地调用各种API，完成复杂任务，就像你在实验室里用各种工具做出精彩的实验一样。
Abstract
Function calling significantly extends the application boundary of large language models, where high-quality and diverse training data is critical for unlocking this capability. However, real function-calling data is quite challenging to collect and annotate, while synthetic data generated by existing pipelines tends to lack coverage and accuracy. In this paper, we present ToolACE, an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data. ToolACE leverages a novel self-evolution synthesis process to curate a comprehensive API pool of 26,507 diverse APIs. Dialogs are further generated through the interplay among multiple agents, guided by a formalized thinking process. To ensure data accuracy, we implement a dual-layer verification system combining rule-based and model-based checks. We demonstrate that models trained on our synthesized data, even with only 8B parameters, achieve state-of-the-art performance on the Berkeley Function-Calling Leaderboard, rivaling the latest GPT-4 models. Our model and a subset of the data are publicly available at https://huggingface.co/Team-ACE.