APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

Recent advances in large language models (LLMs) such as GPT-3, Llama, and their derivatives have significantly improved single-turn understanding and generation capabilities. However, multi-turn human-AI interaction remains challenging due to the complexity of maintaining context, ensuring logical consistency, and executing domain-specific tasks. Prior efforts like Tool-Use Agents, ToolDial, and InterCode have introduced tool invocation and multi-turn reasoning, but often lack systematic validation or high-quality data sources. Manual data collection is costly and limited in scope, restricting model robustness in real-world scenarios. Synthetic data generation has emerged as a promising solution, with methods like MAG-V, BUTTON, and MAGNET attempting to simulate multi-agent interactions, but often suffer from verification issues and limited diversity. The gap between static dataset creation and dynamic, verifiable multi-turn interaction modeling remains a key obstacle.

The core challenge addressed by this work is generating high-quality, diverse, and verifiable multi-turn interaction data for training robust AI agents. Existing datasets are either manually annotated, limiting scalability, or synthetically generated without rigorous validation, leading to low fidelity and poor generalization. Multi-turn interactions involve complex dependencies, API invocation accuracy, and natural dialogue flow, which are difficult to simulate and verify systematically. Without reliable data, models struggle with long-term dependencies, logical coherence, and task completion in multi-turn scenarios. The problem is further compounded by the need for scalable, automated validation mechanisms that can ensure data quality without extensive human intervention, enabling large-scale training of reliable multi-turn agents.

This paper introduces a groundbreaking two-phase framework—APIGen-MT—that combines blueprint validation with simulated interaction. Key innovations include: 1) a systematic blueprint generation process incorporating API dependency graphs, domain policies, and user personas, validated through multi-stage automated checks and committee review; 2) a reflection-based mechanism that iteratively refines task blueprints based on validation feedback; 3) a simulation engine that generates realistic multi-turn dialogues grounded in verified blueprints, ensuring both diversity and correctness; 4) training of a scalable model series (xLAM-2) that outperforms current SOTA models in multi-turn benchmarks. These innovations collectively address the validation bottleneck and enable the scalable creation of high-fidelity multi-turn data.

• �� Phase 1 begins with context preparation, assembling relevant APIs, policies, and user personas.
• �� An LLM-based data generator samples task configurations, including user instructions, groundtruth actions, and expected outputs.
• �� Proposed configurations undergo format validation, environment execution checks (API validity, argument correctness), and policy compliance testing.
• �� A review committee of multiple LLM judges evaluates semantic coherence, completeness, and task sensibility, using majority voting.
• �� Failures trigger a reflection mechanism that summarizes issues and guides iterative refinement of blueprints.
• �� Phase 2 uses validated blueprints to simulate multi-turn interactions, where a human-like agent and environment respond turn-by-turn, producing dialogue trajectories.
• �� Trajectories are validated against groundtruth actions and outputs, ensuring they meet task goals and environmental constraints.
• �� Only verified trajectories are included in the training dataset, ensuring high fidelity and diversity.

The experimental setup involves training models across different scales (1B, 8B, 32B, 70B parameters) using synthetic data generated via the proposed pipeline. The τ-bench and BFCL benchmarks evaluate multi-turn reasoning, function calling, and consistency. Metrics include task success rate, dialogue coherence, API invocation accuracy, and stability across multiple runs. Baselines include GPT-4o and Claude 3.5. Ablation studies compare models trained with and without the validation-refinement pipeline, demonstrating the effectiveness of the verification mechanism. Hyperparameters such as learning rate, batch size, and training steps follow standard practices, with additional tuning for multi-turn tasks. Results show that models trained on verified data outperform baselines, especially in multi-turn and long-horizon scenarios, confirming the pipeline’s robustness.

The models trained with APIGen-MT data achieved scores of 78.2 (τ-bench) and 71.5 (BFCL), surpassing GPT-4o (72.8) and Claude 3.5 (64.3). The 8B model outperformed larger models in multi-turn tasks, validating the verification-simulation approach. Synthetic data quality was confirmed by high task success rates and consistency across multiple trials. Ablation experiments revealed that the validation and reflection steps significantly improved model robustness, reducing error rates and enhancing long-term dependency tracking. The models demonstrated strong generalization across diverse scenarios, indicating the pipeline’s effectiveness in producing reliable multi-turn datasets.

This framework enables scalable generation of multi-turn dialogue datasets for various industries, including customer service, virtual assistants, and enterprise automation. It reduces reliance on manual annotation, accelerates model development, and improves interaction quality. The synthetic data can be used to fine-tune existing models or train new ones, especially in domain-specific applications like finance, healthcare, and logistics. Additionally, the approach facilitates testing and benchmarking of multi-turn systems, supporting industry standards and regulatory requirements. Long-term, integrating multimodal inputs and expanding to unstructured environments could further enhance its utility, driving broader adoption of reliable AI agents.

The pipeline relies on predefined APIs and domain policies, limiting applicability to environments with flexible or unstructured interfaces. Computational costs for large-scale training and data synthesis are high, posing resource challenges. The validation committee, although effective, introduces subjective biases, necessitating further automation. Extending to multimodal or unstructured data remains non-trivial, requiring additional research. Future work should focus on reducing resource demands, automating validation, and broadening environment coverage to improve generalization.