The Bitter Lesson of Tool Calling
This study compares programmatic tool calling (PTC) with native JSON calls across 14 models on BFCL v4, showing a 10.6% accuracy boost for PTC, demonstrating robustness and scalability.
Key Findings
Methodology
This research employs a comprehensive empirical evaluation framework, testing 14 distinct language models across BFCL v4 benchmark tasks. The core comparison involves two paradigms: traditional JSON-based tool calls and programmatic calls where models generate executable Python scripts. The scripts invoke external tools via typed Python stubs, executed in subprocesses to obtain results in a single turn, reducing interaction rounds. The evaluation includes three ablation studies focusing on chaining, parallel fan-out, and context overload scenarios, designed to probe the limitations of JSON calls. Accuracy metrics are computed as the fraction of correctly invoked functions with proper arguments, with confidence intervals ensuring statistical significance. All models are run at temperature zero to guarantee deterministic outputs, enabling precise comparison.
Key Results
- Across all models, 11 out of 14 matched or exceeded baseline accuracy with programmatic calls, with GPT-5.6 models achieving a maximum absolute increase of 10.6%, reaching 84.1% accuracy. The five Anthropic models uniformly matched or surpassed baseline performance, indicating strong capability transfer. Older GPT models, such as GPT-4o, GPT-4.1, and gpt-4o, underperformed due to syntax errors caused by newline encoding issues in multiline scripts. In chain-of-thought tasks, programmatic calls outperformed JSON calls significantly at chain lengths ≥12, with an 18.8% absolute accuracy gap. In high fan-out scenarios (N≥70), programmatic calls maintained 100% enumeration accuracy, whereas JSON calls degraded rapidly, revealing structural limitations. Under context overload conditions, programmatic calls remained stable with only a 2.3% average accuracy drop, while JSON calls suffered a 32% decline, demonstrating superior robustness.
- The results underscore that programmatic tool calling leverages code generation to overcome JSON's rigidity, enabling scalable, reliable multi-step and concurrent tool invocation. It aligns with the model's evolving capabilities, especially in recent generations, and offers a promising pathway for deploying complex AI systems in real-world applications.
Significance
This work highlights a paradigm shift in how large language models interact with external tools. By replacing rigid JSON calls with executable scripts, models can perform multi-step reasoning, handle high concurrency, and adapt to complex workflows more effectively. This approach addresses longstanding bottlenecks in scalability and robustness, paving the way for more autonomous, flexible AI agents capable of managing intricate tasks across diverse domains. The findings are highly relevant for industry applications such as intelligent assistants, automated data analysis, and multi-API orchestration, where efficiency and reliability are paramount. Moreover, the study provides a framework for tracking model capability improvements across generations, guiding future development and deployment strategies.
Technical Contribution
The paper introduces a novel framework where tool invocation is expressed as typed Python scripts, replacing the traditional JSON interface. Key innovations include: • Embedding function schemas as typed Python stubs, allowing models to generate scripts that directly call APIs; • Executing scripts in subprocesses to obtain results in a single turn, reducing interaction overhead; • Designing three targeted ablation studies—chaining, parallelism, and context overload—to rigorously evaluate robustness. This approach leverages the model’s code generation strengths, enabling it to handle complex multi-step workflows and high concurrency without structural limitations inherent in JSON. The framework demonstrates improved scalability, error resilience, and capability tracking across model generations, setting a new standard for tool integration in large language models.
Novelty
This research is the first comprehensive, large-scale comparison of programmatic versus JSON-based tool calling across multiple model generations under standardized benchmarks. Its novelty lies in: • Introducing a typed Python script-based paradigm that allows models to generate executable code for tool invocation; • Systematically evaluating performance across diverse, complex scenarios including chaining, parallel calls, and context overload; • Demonstrating that recent model generations (e.g., GPT-5.6) can effectively utilize script-based calls, achieving significant accuracy improvements. Unlike prior works limited to coding tasks or isolated API calls, this study validates the paradigm’s effectiveness in real-world, multi-step, multi-API environments.
Limitations
- Some models, particularly GPT-4.1 and gpt-4o, exhibit issues with newline encoding in multiline scripts, leading to syntax errors and reduced effectiveness of programmatic calls. This indicates a gap in the models’ code generation robustness, especially for complex scripts.
- The current implementation relies on subprocess execution, which introduces computational overhead and may limit real-time applicability in latency-sensitive scenarios.
- Under extreme context overloads (e.g., 128 schemas), models still show performance fluctuations, suggesting the need for improved context management and schema filtering techniques.
Future Work
Future directions include enhancing models’ code generation fidelity, especially for multiline scripts, by integrating better tokenization and formatting strategies. Developing adaptive script optimization and dynamic type inference could further improve robustness. Extending the paradigm to multi-modal inputs, multi-task workflows, and real-world API ecosystems will be crucial for practical deployment. Additionally, integrating reinforcement learning to optimize tool invocation policies and exploring hybrid paradigms combining JSON and script-based calls could yield even greater flexibility and efficiency. Overall, advancing these areas will help realize autonomous, scalable, and reliable AI agents capable of complex reasoning and multi-tool coordination.
AI Executive Summary
The rapid evolution of large language models (LLMs) has transformed the landscape of artificial intelligence, enabling systems that can understand, generate, and reason across diverse domains. A key aspect of this progress is the ability of models to invoke external tools and APIs, extending their capabilities beyond pure language understanding. Traditionally, tool invocation has relied on structured JSON objects, which, while effective for simple tasks, pose significant limitations when scaling to complex, multi-step workflows involving multiple APIs or high concurrency.
This paper introduces a paradigm shift: programmatic tool calling (PTC). Instead of emitting rigid JSON calls, models generate executable Python scripts that invoke tools via typed stubs, executed in subprocesses to produce results within a single turn. This approach leverages the model’s code generation strengths, allowing for natural chaining, parallelization, and handling of complex argument serialization. The authors systematically evaluate PTC against traditional JSON calls across 14 models spanning from late 2024 to mid-2026, using the BFCL v4 benchmark, which covers a broad spectrum of real-world tasks.
The evaluation reveals that PTC consistently outperforms JSON calls in accuracy, robustness, and scalability. Notably, the GPT-5.6 family achieves a 10.6% absolute improvement, reaching 84.1% accuracy. In complex scenarios such as long chains (≥12 steps), PTC surpasses JSON by 18.8%. Under high fan-out (N≥70), PTC maintains 100% enumeration accuracy, while JSON calls degrade rapidly. Additionally, in context overload experiments, PTC remains stable with only a 2.3% accuracy decline, whereas JSON calls drop by 32%, demonstrating superior robustness.
These findings underscore the importance of script-based invocation, which aligns with the model’s inherent coding capabilities, enabling more flexible, scalable, and reliable tool integration. The research also highlights the influence of model generation quality on performance, noting that newer models like GPT-5.6 are better suited for PTC, whereas older models struggle with multiline script syntax due to encoding issues.
The implications are profound: adopting programmatic calls can significantly enhance the deployment of AI agents in real-world applications such as automated data analysis, multi-API orchestration, and autonomous reasoning. Future work will focus on improving code generation fidelity, extending to multi-modal workflows, and optimizing execution efficiency. Overall, this study paves the way for more intelligent, autonomous, and scalable AI systems, marking a critical step toward realizing the full potential of large language models in complex, real-world tasks.
Deep Analysis
Background
The development of large language models (LLMs) like GPT-4, GPT-5, and Claude series has revolutionized natural language processing, enabling applications from chatbots to automated reasoning systems. A crucial capability is tool invocation, where models call external APIs to extend functionalities such as search, calculation, or data retrieval. Early approaches relied on structured JSON objects to specify function calls, as seen in frameworks like API-Bank and T-Eval, which provided standardized benchmarks for accuracy. However, JSON-based invocation faces scalability challenges: it requires multiple interaction turns for multi-step tasks, struggles with high concurrency (fan-out), and is limited in expressing complex arguments or chaining. Recent research (Wang et al., 2024; Lumer et al., 2026) demonstrated that code actions outperform JSON calls in multi-tool scenarios, leveraging models’ code generation abilities. Despite these advances, there remains a lack of systematic evaluation comparing script-based invocation with traditional JSON methods across diverse models and complex tasks, especially under real-world conditions.
Core Problem
The core challenge addressed in this work is the structural and scalability limitations of JSON-based tool invocation in large language models. Specifically, JSON calls are inherently rigid, requiring multiple interaction turns for sequential tasks, which increases latency and error accumulation. High fan-out scenarios, common in multi-API workflows, cause models to omit calls or produce incomplete results, degrading overall accuracy. Additionally, JSON serialization of arguments becomes cumbersome with complex data types, limiting the expressiveness needed for sophisticated tasks. These bottlenecks hinder the deployment of AI agents capable of autonomous multi-step reasoning and multi-API coordination in real-world applications such as virtual assistants, automated data pipelines, and multi-modal reasoning systems.
Innovation
This paper introduces a novel paradigm—programmatic tool calling (PTC)—that replaces JSON calls with executable Python scripts. Key innovations include: • Embedding function schemas as typed Python stubs, enabling models to generate scripts that directly invoke tools with proper argument serialization; • Executing scripts in subprocesses to obtain results in a single turn, reducing interaction overhead; • Designing targeted ablation studies to evaluate robustness in chaining, parallelism, and context overload scenarios. This approach leverages the natural coding capabilities of models, allowing for seamless chaining of multiple calls, parallel execution, and handling complex argument structures. It addresses the scalability and robustness issues of JSON calls, providing a flexible, efficient, and reliable framework for tool invocation in large models.
Methodology
- �� 定义任务：每个任务包括用户请求、函数接口（带类型签名）和真实调用集，模型需输出正确调用。• 两范式：JSON调用通过结构化JSON定义API，模型输出调用对象；程序化调用导出类型化Python存根，模型生成脚本，调用API函数。• 脚本设计：模型写入Python脚本，导入存根，调用函数，输出结果。• 执行机制：脚本在子进程中运行，输出被解析，无需多轮交互。• 实验设计：包括链式（多步骤调用）、并行（fan-out）和上下文干扰（Schema overload）场景，评估两范式在复杂任务中的表现。• 评估指标：准确率（调用正确且参数匹配），结合置信区间确保统计显著。• 模型设置：所有模型在温度0下运行，确保输出的确定性。
Experiments
采用BFCL v4基准，涵盖8个任务类别，总计309个样本。每个样本定义用户请求、函数接口和正确调用。模型在两范式下生成调用，评估准确率。三项消融实验：链式（链长2-20）、并行（fan-out 7-48）、上下文干扰（Schema overload，128个Schema）。模型包括14个版本，从2024年11月至2026年7月发布。所有模型在温度0下运行，确保输出一致。指标包括：• 调用正确率：所有调用参数正确；• 置信区间：95% Wilson区间；• 失败处理：语法或运行错误判为0。结果通过对比不同模型、场景，验证PTC的优越性。
Results
- �� 在BFCL v4中，11个模型的程序化调用准确率优于或等于JSON调用，GPT-5.6系列模型提升10.6%，最高达84.1%；• 在链式任务中，链长≥12时，程序化调用比JSON高出18.8%；• 在高fan-out（N≥70）场景，程序化调用保持100%枚举准确率，而JSON调用在超过70时开始掉落，显示结构性限制；• 在Schema overload的上下文干扰中，程序化调用仅下降2.3%，而JSON调用下降32%，表现出更强鲁棒性。整体而言，程序化调用在复杂、多任务、多并发环境中表现出更优性能，验证其作为新范式的潜力。
Applications
- �� 立即应用：在企业智能助手、自动化问答中，采用程序化调用提升多步骤、多API集成效率；• 长远愿景：推动大模型在自主推理、多模态、多任务中的应用，构建更智能、更自主的系统。未来，将结合强化学习和自我监督技术，优化脚本生成策略，扩展到多模态、多任务环境，推动AI系统的自主性和智能化水平提升。
Limitations & Outlook
- �� 部分模型（如GPT-4.1、gpt-4o）在多行脚本编码中出现换行符编码问题，导致语法错误，影响效果；• 依赖子进程执行，存在计算开销，影响实时性；• 在极端上下文负载（128 Schema）下，模型表现仍有波动，需优化上下文管理和Schema筛选策略。未来应加强模型代码生成能力，提升脚本执行效率，探索更高效的上下文处理方法。
Plain Language Accessible to non-experts
想象你在厨房做饭，传统的方法就像用一份死板的食谱，每次都得严格按照步骤操作，不能随意调整。而程序化工具调用就像有个聪明的厨师，他可以自己写出一份“食谱”，然后自己动手做菜，一气呵成。这位厨师可以同时准备多道菜，也能根据不同的食材灵活变换做法。这样一来，不仅效率更高，还能做出更复杂的菜肴。研究中，科学家让模型像这个厨师一样，写脚本调用外部工具，避免了用JSON表达的限制，提升了处理复杂任务的能力。就像厨房里的厨师变得更聪明、更灵活一样，模型也变得更强大、更智能。
ELI14 Explained like you're 14
你知道吗？大模型就像一个超级厉害的机器人助手，它可以帮你写作、回答问题，还能调用各种工具，比如查天气、做数学题。以前，它调用工具就像用一个信封，把所有信息装进去，然后寄出去，等待回复。这虽然简单，但太慢，还容易出错。现在，科学家们发明了一个新办法，就像让机器人自己写一段小程序，把所有步骤都写在一块，然后自己跑起来，直接得到答案。这叫程序化调用，它让模型变得更聪明、更快，也更能应对复杂的任务。未来，这样的机器人助手会变得更厉害，帮我们做更多事情！
Abstract
Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been conducted. In this work, we empirically compare programmatic tool calling (PTC) to native JSON tool calling across 14 language models on BFCL v4. In the programmatic tool calling paradigm, tools are exposed as typed Python stubs that the model invokes through code, with execution and results handled in a single agent turn. Programmatic tool calling matches or exceeds native JSON tool calling in 11 of 14 models on BFCL v4, with the GPT-5.6 family achieving a 10.6% improvement over the JSON tool calling baseline. Further, it matches or outperforms baseline in 13 of 14 models under parallel fan-out, and holds stable under context rot conditions where baseline degrades 2.3% on average. Our results demonstrate that programmatic tool calling is a viable and robust alternative to JSON tool calling, with performance tracking model capability across release generations.
References (8)
HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents
Yaxin Du, Yifan Zhou, Yujie Ge et al.
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury et al.
Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios
Shijue Huang, Wanjun Zhong, Jianqiao Lu et al.
Executable Code Actions Elicit Better LLM Agents
Xingyao Wang, Yangyi Chen, Lifan Yuan et al.
The Deterministic Horizon: When Extended Reasoning Fails and Tool Delegation Becomes Necessary
Dongxing Guo, Jikun Wu, S. Yiu
T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step
Zehui Chen, Weihua Du, Wenwei Zhang et al.
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
Tongliang Liu, Cheng Qian, Matej Cief et al.
CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions
Tamer Alkhouli, Katerina Margatina, James Gung et al.