Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe
EDGE framework synthesizes tool-calling data using dynamic graphs, enhancing performance.
Key Findings
Methodology
EDGE framework synthesizes tool-calling data using dynamic graphs, verifying each tool's dependencies through live execution. Fine-tuned with GRPO, the 9B model significantly improves performance on KOPA-BENCH and BFCL benchmarks.
Key Results
- On KOPA-BENCH, the fine-tuned 9B model improved pass@1 by 10 percentage points, nearly matching the performance of the untuned 27B model.
- On BFCL benchmark, the fine-tuned model showed over 5 percentage points improvement in multi-turn tasks.
- Trajectory diversification led to an 11 percentage points increase in pass@4.
Significance
The study addresses the underperformance of open-source models in multi-step tool-calling, providing a new benchmark and data synthesis method, advancing the application of open-source models under data sovereignty.
Technical Contribution
EDGE framework offers a new data synthesis method by verifying tool dependencies through live execution, significantly enhancing the performance of small open-source models, nearly matching larger architectures.
Novelty
First to propose execution-verified dynamic graph synthesis, offering a more reliable multi-step task solution compared to existing tool-calling datasets.
Limitations
- EDGE framework relies on live APIs, which may be affected by API changes.
- In some cases, the model may struggle with high-complexity tasks.
Future Work
Future directions include exploring broader API platforms to extend EDGE framework applications and optimizing model generalization across different tasks.
AI Executive Summary
As data sovereignty regulations tighten, public institutions need to deploy open-source local large language models to handle live government APIs. However, existing open-source models underperform in multi-step tool-calling. To address this, the research team introduces the Korean Open Public API Benchmark (KOPA-BENCH) and the EDGE framework. EDGE synthesizes tool-calling data using dynamic graphs, verifying each tool's dependencies through live execution. The fine-tuned 9B model shows significant improvement on KOPA-BENCH and BFCL benchmarks, nearly matching the performance of the untuned 27B model. The study not only enhances model performance but also provides new solutions for open-source model applications under data sovereignty. Future work will extend EDGE framework applications and optimize model generalization across different tasks.
Deep Analysis
Background
With data sovereignty regulations tightening, public institutions need to deploy open-source models locally to handle live government APIs. However, these models underperform in multi-step tool-calling, lacking effective benchmarks to measure performance gaps.
Core Problem
Open-source models underperform in multi-step tool-calling, lacking effective benchmarks to measure performance gaps. Existing datasets fail to effectively handle live endpoint failures and high cardinality handoffs in Korean public APIs.
Innovation
EDGE framework synthesizes tool-calling data using dynamic graphs, verifying each tool's dependencies through live execution. Compared to existing methods, EDGE offers a more reliable multi-step task solution.
Methodology
- �� Build tool dependency graph, verify each tool's dependencies.
- �� Filter effective paths through live execution.
- �� Fine-tune models using GRPO to enhance performance.
Experiments
Experiments use KOPA-BENCH and BFCL benchmarks to evaluate model performance in multi-step tasks. Fine-tuning the 9B model with GRPO significantly improves performance on KOPA-BENCH and BFCL benchmarks.
Results
The fine-tuned 9B model improved pass@1 by 10 percentage points on KOPA-BENCH, nearly matching the performance of the untuned 27B model. On BFCL benchmark, the fine-tuned model showed over 5 percentage points improvement in multi-turn tasks.
Applications
EDGE framework can be used by public institutions to enhance open-source model applications under data sovereignty, improving tool-calling performance and supporting live government APIs.
Limitations & Outlook
EDGE framework relies on live APIs, which may be affected by API changes. In some cases, the model may struggle with high-complexity tasks.
Plain Language Accessible to non-experts
Imagine a kitchen where EDGE acts as a sous-chef, helping the head chef pass ingredients between different pots. Each pot represents a tool, and the ingredients are data. EDGE ensures each pass is accurate, resulting in a perfect dish. This process is like coordinating multiple steps in a kitchen, ensuring each step proceeds smoothly.
ELI14 Explained like you're 14
Imagine you're playing a complex game, and EDGE is your game assistant, helping you pass items between different levels. Each level represents a tool, and the items are data. EDGE ensures each pass is accurate, ultimately helping you beat the game. This process is like coordinating multiple steps in a game, ensuring each step proceeds smoothly.
Glossary
EDGE Framework
A method for synthesizing tool-calling data using dynamic graphs, verifying each tool's dependencies.
Used to enhance open-source model performance in multi-step tasks.
KOPA-BENCH
Korean Open Public API Benchmark, consisting of 145 multi-step tasks.
Used to evaluate open-source model performance in multi-step tasks.
GRPO
A fine-tuning method used to enhance model performance.
Used to fine-tune datasets generated by EDGE framework.
Dynamic Graph
A method for verifying tool dependencies through live execution.
Used to synthesize tool-calling data.
Multi-step Tasks
Tasks requiring multiple tool calls.
EDGE framework is used to enhance performance in multi-step tasks.
Open Questions Unanswered questions from this research
- 1 How can EDGE framework applications be extended to more API platforms?
- 2 How can model generalization across different tasks be optimized?
Applications
Immediate Applications
Public Institution Application
Public institutions can use EDGE framework to enhance open-source model tool-calling performance under data sovereignty.
Long-term Vision
Global API Platform Expansion
EDGE framework can be expanded to more global API platforms, enhancing model generalization across different tasks.
Abstract
Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on the BFCL benchmark.