Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation
Data Turnstile framework generates high-quality function-calling data, boosting Qwen3-0.6B accuracy to 75.9%.
Key Findings
Methodology
Data Turnstile decomposes multi-turn tool-use interactions into constrained stepwise generation with validation and error-feedback loops, providing fine-grained control over API diversity, conversation complexity, and output correctness. The framework uses a Directed Acyclic Graph (DAG) structure to ensure quality at each step.
Key Results
- On the BFCL single-turn benchmark, Qwen3-0.6B fine-tuned on Turnstile data achieves 75.9% accuracy, improving by 8.5 percentage points over the base model with thinking enabled, approaching the larger Qwen3-1.7B and Qwen3-4B models.
- On the τ^2-bench multi-turn benchmark, Turnstile-trained Qwen3-1.7B achieves 31.1% pass rate in the Telecom domain, improving 4.7 times over its base and surpassing the Qwen2.5-32B-Instruct model.
- Turnstile-generated data shows significant diversity and quality improvements over open-source data, particularly in tool-call sequence and structural diversity.
Significance
The Data Turnstile framework provides a method for small language models to generate high-quality training data, addressing the issues of data scarcity and noise, significantly enhancing model performance in function-calling tasks. This research opens new possibilities for deploying language models on edge devices, reducing costs and latency.
Technical Contribution
The framework avoids reliance on large language models through stepwise generation and validation mechanisms, enabling local execution and rapid iteration. It offers a method for generating diverse and high-quality data applicable to various domain API definitions, enhancing small language models' tool-calling capabilities.
Novelty
Data Turnstile is the first to propose decomposing multi-turn interactions into constrained stepwise generation, improving data quality through error-feedback loops, offering significant advantages over existing single-shot generation methods.
Limitations
- The framework faces high failure rates when handling very complex multi-turn interactions, especially with parallel API calls.
- The quality of generated data depends on template design, which may not cover all real-world scenarios.
Future Work
Future research can explore improving success rates for generating complex interactions and extending the framework to support more types of APIs and interaction scenarios.
AI Executive Summary
Data Turnstile is an open-source framework designed to generate high-quality function-calling data, particularly for small language models. Existing small language models face challenges in tool-use tasks due to data scarcity and noise, limiting their performance. Data Turnstile addresses these issues by decomposing multi-turn interactions into constrained stepwise generation, combined with validation and error-feedback loops for precise control over generation quality.
Experimental results demonstrate that small language models fine-tuned with Turnstile data perform exceptionally well on BFCL and τ^2-bench benchmarks, with significant improvements in accuracy and pass rates, even surpassing larger models. This framework offers new possibilities for deploying small language models on edge devices, reducing costs and latency while enhancing data privacy.
Nevertheless, the framework faces challenges in handling very complex interactions. Future research can explore improving success rates for generating complex interactions and extending the framework to support more types of APIs and interaction scenarios.
Deep Analysis
Background
With the evolution of language models, small language models have gained attention due to their advantages in low latency, cost, and data privacy. However, they face challenges in tool-use tasks due to data scarcity and noise, limiting their performance. Large language models can compensate for low-quality supervision through sheer capacity, whereas small models require high-quality data support.
Core Problem
Small language models face challenges in tool-use tasks due to data scarcity and noise, unable to compensate for low-quality supervision like large models. Data quality becomes the critical bottleneck for training small language models, necessitating a method to generate high-quality training data.
Innovation
Data Turnstile framework decomposes multi-turn interactions into constrained stepwise generation, combined with validation and error-feedback loops to improve data quality. It uses a Directed Acyclic Graph (DAG) structure to ensure quality at each step and supports local execution and rapid iteration.
Methodology
- �� Decompose multi-turn interactions into a Directed Acyclic Graph (DAG) structure, defining roles such as user queries, reasoning traces, API calls, execution outputs, and assistant responses.
- �� Independently generate each role with validation and error-feedback loops to improve generation quality.
- �� Achieve data diversity and quality control through template design.
Experiments
Experiments were conducted on BFCL and τ^2-bench benchmarks, using small language models of different scales for fine-tuning. Evaluation metrics included accuracy and pass rates, comparing the impact of different data sources and generation methods.
Results
Qwen3-0.6B model fine-tuned with Turnstile data achieves 75.9% accuracy on BFCL single-turn benchmark and 31.1% pass rate on τ^2-bench multi-turn benchmark, significantly outperforming larger models.
Applications
The framework is applicable for generating data for various domain API definitions, enhancing small language models' tool-calling capabilities, suitable for edge device deployment, reducing costs and latency.
Limitations & Outlook
The framework faces high failure rates when handling very complex multi-turn interactions, especially with parallel API calls. The quality of generated data depends on template design, which may not cover all real-world scenarios.
Plain Language Accessible to non-experts
Imagine a kitchen where Data Turnstile acts like a smart chef assistant. It prepares ingredients step-by-step according to the recipe, ensuring each step is accurate. If something goes wrong, it corrects it promptly, ensuring the final dish's quality. This way, even a small kitchen can produce delicious meals.
ELI14 Explained like you're 14
Imagine you're playing a game, and Data Turnstile is like a super helper guiding you through each task. It checks if each step is correct, and if there's a mistake, it helps you fix it. This way, you can easily win the game! Isn't that cool?
Glossary
API (Application Programming Interface)
An API is an interface between applications, allowing them to exchange data and functions.
In the paper, APIs are the core objects for data generation.
DAG (Directed Acyclic Graph)
A DAG is a graph structure without cycles, suitable for representing dependencies.
Used to decompose multi-turn interactions into stepwise generation.
CoT (Chain of Thought)
CoT is a reasoning method that solves problems by thinking step-by-step.
Used to evaluate the model's reasoning ability.
SLM (Small Language Model)
SLM is a language model with fewer parameters, suitable for edge devices.
Research subject facing data scarcity issues.
BFCL (Berkeley Function Calling Leaderboard)
BFCL is a benchmark for evaluating models' tool-calling abilities.
Used to assess the framework's data generation effectiveness.
Open Questions Unanswered questions from this research
- 1 How to further improve success rates for generating complex interactions?
- 2 How to extend the framework to support more types of APIs?
Applications
Immediate Applications
Edge Device Deployment
Small language models can be deployed on edge devices, reducing costs and latency.
Customer Service Applications
High-quality data can be used to train customer service bots, improving response accuracy.
Long-term Vision
Cross-Domain Tool Use
The framework can be extended to more domains, supporting complex tool-use scenarios.
Abstract
Small language models (SLMs) are attractive for agentic deployment due to low latency, reduced cost, and on-device privacy, yet they struggle with tool-use tasks where training data is scarce and noisy. Unlike larger models, SLMs cannot compensate for low-quality supervision through sheer capacity, making data quality the critical bottleneck. We present Data Turnstile, an open-source framework that takes user-defined API specifications and generates high-quality synthetic training data for function calling. Turnstile decomposes multi-turn tool-use interactions into constrained, stepwise generation with validation and error-feedback loops, providing fine-grained control over API diversity, conversation complexity, and output correctness. We demonstrate effectiveness of domain adaptation with Turnstile data on two challenging function calling benchmarks. On the BFCL single-turn benchmark, a Qwen3-0.6B fine-tuned on Turnstile data without chain-of-thought achieves 75.9% overall accuracy (versus 67.4% for the base model with thinking enabled), closing the gap with thinking-enabled Qwen3-1.7B (78.4%) and Qwen3-4B (79.9%) despite being 3$\times$ and 7$\times$ smaller respectively. On $τ^2$-bench, a multi-turn agentic benchmark, Turnstile-trained Qwen3-1.7B achieves 31.1% pass^1 on the Telecom domain, improving 4.7$\times$ over its 6.6% base and surpassing Qwen2.5-32B-Instruct (27.4%), a model 19$\times$ larger. Turnstile-trained Qwen3-0.6B achieves 24.6%, improving 7$\times$ over its 3.5% base and approaching the 32B model (53$\times$ larger). We release Data Turnstile along with a dataset spanning 1,000+ APIs and 100K+ multi-turn interactions.