ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
ToolSandbox is a stateful, conversational benchmark for evaluating LLM tool use, emphasizing implicit state dependencies and dynamic milestone tracking.
Key Findings
Methodology
ToolSandbox integrates stateful tool execution, implicit state dependencies, a built-in user simulator, and a dynamic evaluation strategy. It models tools as Python functions within a simulated environment, supporting multi-turn dialogues and milestone-based assessment. The framework employs a message bus for communication among user, agent, and environment, enabling realistic interaction scenarios. It evaluates models across 1032 scenarios, measuring tool invocation accuracy, dialogue coherence, and error detection via similarity metrics and milestone matching, providing a comprehensive view of model capabilities in complex tasks.
Key Results
- The top-performing GPT-4o model achieved an average similarity score of 73.0 across scenarios, significantly outperforming open-source models like Hermes at 31.4. Tasks involving state dependencies and insufficient information remain challenging, with models frequently making errors in nested state management and canonicalization, especially in time and location reasoning.
- In complex scenarios with multiple tools and nested dependencies, larger models like GPT-4 and Claude-Opus showed performance drops due to parallel tool invocation errors. Introducing milestone-based evaluation improved accuracy, with milestone matching exceeding 85%, validating the framework’s robustness.
- Ablation studies indicated that incorporating knowledge boundaries and demonstration prompts in the user simulator reduced error rates, highlighting the importance of prompt design in interactive evaluation. The results underscore the need for more sophisticated state management and reasoning capabilities in future models.
Significance
This work advances the evaluation paradigm by enabling multi-turn, state-aware, and interactive assessment of LLMs in realistic tool-using scenarios. It addresses critical gaps in existing benchmarks that lack support for implicit state dependencies and dynamic task progression. By providing a detailed, scalable, and interpretable evaluation framework, ToolSandbox facilitates targeted improvements in model reasoning, tool invocation, and dialogue management, fostering progress toward autonomous, reliable AI agents capable of complex real-world tasks.
Technical Contribution
Key innovations include modeling implicit state dependencies among tools, implementing a milestone-based dynamic evaluation system, and developing a simulated user that interacts naturally within multi-turn dialogues. The framework’s modular design allows flexible scenario creation and detailed error analysis, supporting ablation studies on prompt strategies and tool configurations. These contributions significantly enhance the granularity and realism of LLM evaluation, setting new standards for assessing multi-modal, multi-step reasoning capabilities.
Novelty
This is the first benchmark to incorporate implicit state dependencies, multi-turn conversational evaluation, and milestone-based trajectory analysis in a unified framework. Unlike prior static or single-turn benchmarks, ToolSandbox captures the complexity of real-world tool use, including nested dependencies and error correction, providing a more faithful measure of LLM competence in dynamic environments. Its comprehensive design bridges gaps between static evaluation and real interactive scenarios.
Limitations
- The benchmark relies on predefined scenarios and tools, limiting coverage of unforeseen real-world situations. Future work should incorporate adaptive scenario generation and broader toolsets.
- Models still struggle with nested state dependencies and multi-tool coordination, indicating the need for improved reasoning and state tracking architectures.
- Evaluation metrics based on similarity and milestone matching may not fully reflect user satisfaction or practical utility, suggesting integration of user-centric metrics in future iterations.
Future Work
Future directions include expanding scenario diversity, integrating reinforcement learning to improve state management, and incorporating multi-modal inputs for richer context understanding. Enhancing the user simulator’s realism and extending evaluation to real-world deployments will further bridge the gap between benchmark performance and practical AI applications.
AI Executive Summary
ToolSandbox represents a significant leap forward in evaluating large language models’ tool use capabilities within realistic, multi-turn, and state-dependent scenarios. Traditional benchmarks have largely focused on static, single-turn tasks, which fail to capture the complexities of real-world interactions involving implicit state dependencies, nested tool calls, and dynamic task progression. In response, ToolSandbox introduces a comprehensive framework that models tools as Python functions within a simulated environment, supported by a built-in user simulator and milestone-based evaluation metrics.
The core innovation lies in supporting implicit state dependencies—where tools depend on the world state, which the agent must infer and manipulate without explicit instructions—and in enabling multi-turn dialogues that reflect real user interactions. The framework employs a message bus for communication, allowing flexible, multi-round exchanges between user, agent, and environment. It also incorporates a dynamic evaluation strategy based on milestones and Minefields, which define critical events that must or must not occur, providing a nuanced measure of task success.
Experimental results across 1032 scenarios reveal that while models like GPT-4o excel, significant gaps remain, especially in tasks involving complex state dependencies and canonicalization. The framework’s detailed error analysis and ablation studies demonstrate that prompt design, knowledge boundaries, and demonstration prompts substantially influence performance. These insights underscore the importance of sophisticated reasoning and state management in future AI systems.
Overall, ToolSandbox offers a scalable, interpretable, and realistic benchmark that bridges the gap between static evaluation and real-world AI deployment. Its comprehensive design and promising results pave the way for developing more autonomous, reliable, and context-aware language agents capable of complex tool use, with broad implications for AI research and industry applications.
Despite its strengths, the framework faces limitations such as scenario coverage and model reasoning in nested dependencies. Future work will focus on expanding scenario diversity, improving reasoning architectures, and integrating multi-modal data to enhance robustness and applicability, ultimately guiding the evolution of intelligent, interactive AI systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭，你有很多厨具，比如刀、锅、碗。这些厨具就像工具箱里的工具，你需要用它们来做不同的菜。有时候，你会忘记放盐，或者锅还没热好。这个系统就像是一个聪明的厨师助手，它能记住你用过的厨具状态，比如锅是不是热了，盐是不是放了。它还能帮你规划做菜的步骤，比如先把菜洗干净，再炒一炒。它会和你一直对话，提醒你下一步该做什么，或者告诉你哪里出错了。这样，你就不用担心忘记步骤或者用错厨具了。它让做饭变得更简单、更有条理，就像有个聪明的厨师在帮你管理厨房，让你轻松做出美味的菜肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你要完成很多任务，比如找到宝藏、打败怪兽，还要用不同的道具。有时候你会忘记用哪个道具，或者不知道下一步怎么走。ToolSandbox就像是你的游戏助手，它能记住你用过的道具状态，比如你是不是拿到了钥匙，门是不是开着。它还能帮你规划下一步，比如告诉你需要用火箭发射器才能到达那个地方。它会和你一直对话，帮你解决所有难题，让你玩得更开心、更顺利。就像有个聪明的朋友一直在你身边，帮你解决所有的难题，让你变得更厉害！
Abstract
Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox