Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Proposed CDD detects data contamination via output distribution peakedness, improving detection accuracy by 21.8%-30.2%.
Key Findings
Methodology
This paper introduces CDD, which analyzes the peakedness of sampled model outputs' edit distance distribution to detect data contamination. It models the distribution by computing pairwise edit distances among samples, using the peak as an indicator of contamination. The approach requires only sampled texts, not access to model probabilities or training data. Two benchmarks, DETCON and COMIEVAL, are constructed to evaluate detection and mitigation effectiveness. Extensive experiments show CDD outperforms existing methods with 21.8%-30.2% improvements in accuracy, F1, and AUC. Additionally, TED is proposed to correct output distributions, significantly reducing performance inflation caused by contamination, up to 66.9%. The methods are validated on multiple models, including ChatGPT, revealing high contamination risks on datasets like HumanEval.
Key Results
- CDD achieves 21.8%-30.2% higher accuracy and AUC than baselines across detection tasks, especially effective for implicit contamination. The peakness metric reliably indicates contamination levels.
- TED effectively mitigates performance overestimation, reducing artificial gains by up to 66.9%, leading to more trustworthy evaluations.
- In real-world scenarios, ChatGPT shows signs of data contamination on HumanEval, emphasizing the importance of detection and correction methods.
Significance
This work addresses the critical challenge of detecting and mitigating data contamination in large language models, which affects their evaluation credibility and deployment safety. By enabling only output-based detection, it overcomes the limitations of opaque training data and model black-box nature, fostering more reliable AI systems.
Technical Contribution
The paper pioneers a distribution-based contamination detection method that relies solely on sampled outputs, avoiding dependence on model internals. It introduces the peakedness measure of edit distance distribution, providing a novel statistical indicator for contamination. The construction of specialized benchmarks further advances evaluation standards in this domain.
Novelty
This is the first approach to detect data contamination solely through output distribution analysis without requiring access to training data or model probabilities. The peakedness metric offers a new perspective, outperforming traditional n-gram or embedding similarity methods, especially for implicit contamination detection.
Limitations
- The computational cost of calculating edit distances for large sample sets can be high, limiting scalability.
- Threshold parameters like ξ and α require empirical tuning and may not generalize across all models or datasets.
- Effectiveness in multi-modal or highly complex contamination scenarios remains to be validated.
Future Work
Future research will focus on optimizing computational efficiency, developing adaptive thresholding mechanisms, and extending detection to multi-modal models. Exploring integration with automated training data auditing and real-time contamination monitoring are promising directions.
AI Executive Summary
Large language models (LLMs) have revolutionized NLP and AI applications, yet their training data often contain leaks and synthetic contamination, leading to inflated performance metrics. Traditional detection methods rely on access to training data or model internals, which are often unavailable due to privacy and black-box constraints. This paper introduces a novel approach—Contamination Detection via output Distribution (CDD)—which leverages the peakedness of the output text distribution to identify data contamination solely from sampled outputs.
The core idea is that training on contaminated data causes models to produce highly similar outputs, resulting in a peaked edit distance distribution. By modeling this distribution and measuring its peakedness, CDD effectively detects both explicit and implicit contamination. The authors construct two benchmarks, DETCON and COMIEVAL, to evaluate detection accuracy and mitigation effectiveness across various contamination scenarios, including different models, data leak types, and contamination levels.
Experimental results demonstrate that CDD surpasses existing methods with an average improvement of 21.8%-30.2% in accuracy, F1, and AUC metrics. Furthermore, the paper proposes Trustworthy Evaluation via output Distribution (TED), which corrects the output distribution by excluding peakedness and duplicates, significantly reducing performance inflation caused by data leaks—up to 66.9%. In real-world applications, the authors reveal that ChatGPT exhibits high potential for data contamination on the HumanEval benchmark, underscoring the practical importance of these methods.
Overall, this work offers a robust, model-agnostic framework for detecting and mitigating data contamination, fostering more trustworthy and reliable large language models. It opens new avenues for research in model evaluation, safety, and data auditing, addressing a critical bottleneck in AI deployment.
Deep Analysis
Background
随着大规模预训练模型的广泛应用，数据污染问题逐渐成为影响模型可信性的重要因素。早期如GPT-3采用13-gram重叠检测，但面对复杂的隐性污染和非公开数据，效果有限。近年来，研究者提出多种检测方法，如嵌入相似度、困惑度分析等，但都依赖模型内部信息或训练数据访问。随着模型规模不断扩大，synthetic data的生成和数据泄露问题愈发严重，传统方法难以应对。数据污染不仅导致性能虚高，还掩盖模型缺陷，影响实际应用的可靠性。因此，开发无需访问训练数据、能检测隐性污染的检测工具成为研究热点。
Core Problem
核心问题在于如何在模型黑箱和非公开数据环境下，准确检测训练数据中的污染，尤其是隐性污染。现有方法多依赖模型概率或训练集信息，难以识别变体和间接污染。数据污染会导致模型在测试集上的虚假优异表现，影响评估的可信性，甚至误导后续优化。解决这一难题需要新颖的检测机制，能在仅有模型输出文本的情况下，识别潜在污染源。
Innovation
本研究的创新点在于提出基于输出分布峰值的检测方法CDD，利用采样文本的编辑距离分布特征，识别模型输出的异常集中。相比传统n-gram或嵌入相似度，CDD无需访问模型概率或训练集，适应封闭模型环境。结合新构建的DETCON和COMIEVAL基准，系统验证了在多场景、多污染类型下的优越性能。TED则通过校正输出分布，有效缓解污染带来的性能虚高问题，提升评估的可信度。这些创新推动了模型安全检测的理论与实践发展。
Methodology
- �� 采样：从模型获取文本样本。• 编辑距离：计算样本间的Token级Levenshtein距离。• 分布建模：构建输出距离的概率分布ρ(d)。• 峰值指标：计算峰值Peak(M; x)，衡量分布集中程度。• 污染检测：若峰值超过阈值ξ，则判定为污染。• 校正：TED通过排除峰值和去重，校正输出分布，改善评估。• 参数调节：α、ξ、τ等超参数优化检测效果。
Experiments
使用DETCON和COMIEVAL两个基准，模拟多种污染场景，包括不同污染比例、类型（显性/隐性）和模型（CodeLlama、Llama2等）。比较CDD与n-gram、嵌入相似度、困惑度等方法，评估准确率、F1、AUC指标。参数调优后，CDD在检测隐性污染时表现优异，平均提升21.8%-30.2%。TED在不同污染程度下显著降低性能虚高，最高缓解66.9%。实证中，ChatGPT在HumanEval表现出潜在污染风险，验证了方法的实用性。
Results
CDD在多场景下均优于对比方法，检测准确率提升明显，特别在隐性污染中表现出鲁棒性。TED有效校正模型输出，减少虚假性能提升，确保评估可信。实测中，ChatGPT在HumanEval中存在明显污染迹象，验证了检测工具的实用价值。
Applications
可广泛应用于模型训练监控、评估体系优化、模型安全审查等场景。特别适合封闭模型或非公开数据环境，为模型开发者提供污染检测和评估校正工具，提升模型可信度。未来可结合自动化参数调节，实现实时监控。
Limitations & Outlook
方法在超大模型上的计算成本较高，峰值阈值参数依赖经验，泛化能力有限。对多模态污染或极端污染场景的检测效果尚待验证。未来需优化算法效率，扩展多模态检测能力。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂每天生产各种商品。工厂的质量控制员需要确保每个商品都符合标准，但工厂里有些商品可能是被偷工减料的假货。传统方法就像用放大镜检查每个商品，但这很费时间，也不一定能找到所有假货。现在，工厂引入了一种新工具——它会分析所有商品的特征，找出那些看起来特别相似或异常的商品。这个工具就像CDD，它通过观察商品的细节分布，判断是否有假货混入。这样一来，工厂可以更快、更准确地找到问题商品，保证产品质量。这就像模型的输出文本一样，分析它们的分布，判断是否有数据污染，确保模型的“生产线”正常运作。
ELI14 Explained like you're 14
想象你在学校里，有个老师会给你很多作业。正常情况下，你的答案应该多样化，不会每次都一样。但如果你总是抄别人的答案，答案就会变得很像，甚至一模一样。这个老师可以通过观察你的答案，发现你是不是抄袭了。这个检测方法就像CDD，它会看你的答案之间的相似程度。如果答案都很像，说明你可能抄了。这样，老师就能知道你是不是作弊了。这个方法不用看你平时的作业，只看你这次的答案，就能判断你是不是抄袭。它用一种特别的“相似度”指标，帮老师找到那些可能有问题的答案。这样一来，考试就更公平了，大家都得靠自己努力。
Abstract
Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs' training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination. However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs. CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM's output distribution. To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM's output distribution. To facilitate this study, we introduce two benchmarks, i.e., DetCon and ComiEval, for data contamination detection and contamination mitigation evaluation tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8\%-30.2\% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination. TED substantially mitigates performance improvements up to 66.9\% attributed to data contamination across various contamination setups. In real-world applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark.