SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
SynAE employs multi-metric evaluation to assess synthetic data quality for tool-calling agents, focusing on validity, fidelity, and diversity, enabling robust benchmarking.
Key Findings
Methodology
SynAE integrates multiple metrics—Validity Rate, Key Node Dependency, Attribute Match, KNN-Precision, FID—to evaluate synthetic datasets across structure, semantics, and diversity. It employs LLMs as judges for validity, and uses controlled generation schemes (e.g., blank filling, oversampling, in-context generation) to simulate common synthetic flaws. The framework compares real and synthetic data on multiple axes, capturing fine-grained quality variations. Extensive experiments on datasets like T1, BFCL, and ACP demonstrate its sensitivity and robustness, effectively identifying quality degradations under various synthetic generation parameters.
Key Results
- On T1, increasing masking probability p from 0.1 to 0.9 in blank filling caused KNN-Precision for instructions and outputs to drop from near 1 to close to 0, indicating semantic degradation. Similarly, oversampling with higher ratio r reduced output recall significantly, validating the metrics’ sensitivity to diversity loss.
- In industry-standard NeMo synthetic data, SynAE accurately detected quality drops in fidelity and diversity, aligning with manual assessments. The framework’s multi-metric approach provided a comprehensive picture of synthetic data performance.
- Across datasets, SynAE consistently distinguished between high-quality and degraded synthetic data, outperforming single-metric baselines, and offering actionable insights for data augmentation and model evaluation.
Significance
This work addresses a critical gap in synthetic data evaluation, providing a systematic, multi-dimensional framework that enhances reliability and safety in deploying tool-calling agents. It enables practitioners to quantitatively compare synthetic datasets, optimize generation processes, and ensure model robustness. The integration of structural, semantic, and diversity metrics advances the state-of-the-art, fostering more trustworthy AI systems in real-world applications.
Technical Contribution
The paper introduces a comprehensive evaluation suite combining structural dependency metrics, semantic similarity measures, and diversity scores, unified under a multi-axis framework. The innovative use of LLMs as judges for validity, coupled with controlled synthetic flaw simulations, enhances assessment granularity. The framework’s extensibility across datasets and generation schemes marks a significant step forward in synthetic data quality control, bridging gaps between theoretical metrics and practical deployment needs.
Novelty
This is the first comprehensive multi-metric evaluation framework tailored for multi-turn, tool-calling agent trajectories, integrating structural, semantic, and diversity assessments. The use of LLMs as judges for validity and the simulation of common synthetic flaws for sensitivity analysis represent key innovations that surpass prior single-metric or heuristic approaches.
Limitations
- The framework’s sensitivity to extreme semantic errors remains limited; future work could incorporate more advanced language understanding models.
- Computational cost is relatively high, especially for large datasets or real-time evaluation, which may hinder scalability.
- Generalization to new domains requires careful attribute and property selection, posing challenges for broad applicability.
Future Work
Future research will focus on reducing computational overhead, integrating multimodal data, and enabling real-time quality monitoring. Extending the framework to handle more complex multi-task scenarios and diverse data modalities will further enhance its utility. Additionally, exploring unsupervised or self-supervised approaches for attribute discovery and evaluation could make SynAE more adaptable across domains.
AI Executive Summary
The rapid development of large language models and their integration with tool-calling capabilities have revolutionized AI applications, but evaluating the quality of synthetic data used in training and benchmarking remains a challenge. Existing methods often rely on single metrics like accuracy or BLEU scores, which fail to capture the complex multi-turn interactions and multi-modal dependencies inherent in real-world agent behaviors. To address this, SynAE introduces a multi-dimensional evaluation framework that combines structural, semantic, and diversity metrics, providing a comprehensive assessment of synthetic datasets.
By leveraging advanced language models as judges and employing controlled generation schemes—such as blank filling, oversampling, and in-context generation—SynAE can detect subtle quality degradations. Extensive experiments on benchmark datasets like T1, BFCL, and ACP demonstrate its ability to distinguish high-quality synthetic data from flawed samples, correlating well with manual assessments. These results highlight the framework’s robustness and practical relevance, offering a valuable tool for researchers and industry practitioners.
The broader impact of this work lies in enabling safer, more reliable deployment of AI systems. By systematically quantifying synthetic data quality, SynAE facilitates better data augmentation, model evaluation, and risk mitigation. Its flexible design supports future extensions, including real-time monitoring and multi-modal evaluation, promising a significant step forward in AI quality assurance. Despite some limitations in computational efficiency and domain generalization, ongoing developments aim to overcome these challenges, paving the way for more trustworthy AI ecosystems.
Deep Analysis
Background
近年来，大模型在自然语言处理中的应用不断深化，合成数据作为训练和评估的重要资源逐渐普及。传统评估多依赖单一指标，如准确率或BLEU分数，难以全面反映多轮交互、工具调用等复杂场景的质量。学界提出了多种结构化评估框架，但缺乏统一、多维的指标体系，难以应对真实应用中的多样性和复杂性。行业内也逐步认识到合成数据的质量直接影响模型的可靠性和安全性，亟需科学、系统的评估工具。本研究在此背景下，提出了SynAE框架，旨在弥补现有方法的不足，推动合成数据在实际中的广泛应用。
Core Problem
当前合成数据评估多依赖单一指标，无法同时衡量真实性、多样性和实用性。尤其在多轮工具调用场景中，缺乏系统性指标，导致模型性能难以准确反映。现有指标对结构和语义的敏感度不足，无法捕捉细粒度的质量变化。这限制了合成数据在复杂任务中的应用效果。解决这一问题，要求开发多维指标体系，结合结构、语义和多样性，全面评估合成数据的质量，确保其在实际场景中的有效性。
Innovation
本研究的创新点包括：
1）提出多指标融合的评估体系，涵盖结构、语义和多样性，弥补单一指标的不足；
2）引入LLM作为判别者，提升语义理解能力，增强评估的准确性；
3）设计控制生成方案（如空白填充、过采样、上下文生成），模拟常见缺陷，验证指标敏感性；
4）实现跨数据集的适用性，支持多轮、多任务场景的评估。该方法结合结构和语义指标，突破传统评估的局限，为合成数据质量控制提供新思路。
Methodology
- �� 输入：真实数据集和合成数据集，支持缺失响应或工具调用。
- �� 结构指标：利用Key Node Dependency和Attribute Match，衡量指令-响应关系和属性一致性。
- �� 语义指标：采用KNN-Precision、FID，评估文本相似性和分布距离。
- �� 多样性指标：使用Vendi Score和Attribute Diversity，衡量样本间差异。
- �� 控制方案：设计空白填充、过采样、上下文生成和无效化方案，模拟不同缺陷。
- �� 评估流程：结合LLM判别，计算Validity Rate、Fidelity、Diversity，输出多维指标。
Experiments
在T1、BFCL、ACP等公开数据集上，采用行业标准NeMo合成工具，调节参数（如p、r、k）模拟不同缺陷。对比不同合成方案的指标表现，验证SynAE的敏感性和鲁棒性。实验还包括不同模型（GPT-4、Llama）和不同指标组合的效果分析，确保指标的全面性和实用性。
Results
SynAE能准确检测空白填充比例p的变化，KNN-Precision从接近1降至0，反映语义退化。过采样比例r增加，输出KNN-Recall显著下降，验证多样性指标的敏感性。行业工具NeMo生成的合成数据中，SynAE成功识别出质量退化，验证其在实际场景中的应用潜力。整体上，指标体系能细粒度反映合成数据的质量变化，为后续优化提供依据。
Applications
该框架适用于工业界在模型部署前的合成数据质量检测，确保模型在实际环境中的可靠性。可用于自动化监控合成数据生成流程，提升模型安全性和隐私保护。未来，结合实时监控和多模态信息，将实现全流程的质量控制，推动智能系统的安全落地。
Limitations & Outlook
目前指标对极端缺陷的敏感度有限，复杂场景下的多模态评估尚未实现。计算成本较高，限制大规模应用。未来需优化算法效率，扩展多任务、多模态支持，提升指标的普适性和实用性。
Plain Language Accessible to non-experts
想象你在一个工厂里，生产各种玩具。每个玩具都要符合一定的标准，比如外形、颜色和功能。现在，工厂用机器自动制造玩具，但有时候机器会出错，生产出不符合标准的玩具。为了确保玩具质量，工厂需要一套检测系统，能判断每个玩具是否合格。这个检测系统会检查玩具的外观、功能是否正常，还会比较不同玩具的多样性，确保没有太多一样的。SynAE就像这个检测系统，它用多种方法检查合成数据的“质量”，确保这些数据既真实又丰富，能帮助模型更好地学习和测试。它不仅看外表，还会用智能“眼睛”判断内容是否合理，确保合成数据像真实一样，能用在各种复杂场景中。
ELI14 Explained like you're 14
想象你在学校里，老师让你写作文，但你可以用电脑帮忙。你写完后，老师会检查你的作文是不是符合要求，内容是不是丰富，句子是不是多样。现在，很多AI模型也是这样，它们用大量的“作文”来学习。为了确保这些“作文”质量高，科学家们开发了一个叫SynAE的“作文检查器”。它会用不同的标准，比如内容是否合理、句子是否丰富、主题是否多样，来评估这些合成的“作文”。它还能模拟一些写作中的问题，比如内容重复或不符合题意，测试检测器的敏感度。这样，AI模型就能用更高质量的数据学习，变得更聪明、更可靠。就像老师用多种方法检查你的作文一样，SynAE用多维指标确保合成数据的“作文”都符合标准，帮助AI更好地成长。
Abstract
Today, tool-calling agents are commonly evaluated or tested on static datasets of execution traces, including input commands, agent responses, and associated tool calls. However, internal production datasets are often insufficient or unusable for testing; for example, they may contain sensitive or proprietary data, or they may be too sparse to support comprehensive testing (especially pre-deployment). In these settings, practitioners are increasingly replacing or augmenting real datasets with synthetic ones for evaluation purposes. A key challenge is quantifying the relation between these synthetic datasets and the real data. We introduce SynAE, an evaluation framework for assessing how well synthetic benchmarks for multi-turn, tool-calling agents replicate and augment the characteristics of real data trajectories. SynAE assesses the validity, fidelity, and diversity of synthetic data across four metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes. SynAE detects fine-grained variations in data validity, fidelity and diversity, and shows that no single metric is sufficient to fully characterize synthetic data quality, motivating a multi-axis evaluation of synthetic data for agent testing. A demo of SynAE is available at https://synae-2026-synae-demo.static.hf.space/index.html, with code at https://github.com/wsqwsq/SynAE.