Detecting Pretraining Data from Large Language Models
Proposes MIN-K% PROB, a reference-free method for detecting pretraining data, improving 7.4% AUC on WIKIMIA.
Key Findings
Methodology
The approach relies on the hypothesis that unseen examples contain a few outlier tokens with low probabilities, while seen examples do not. It computes the average negative log-likelihood of the lowest K% tokens in a text, using only the model's output probabilities, without training any reference or shadow models. By leveraging Wikipedia timestamp data, a dynamic benchmark is constructed to evaluate detection across models like LLaMA, GPT-Neo, and Pythia. The method is simple, efficient, and applicable in black-box settings, enabling detection without access to training data or additional training.
Key Results
- On WIKIMIA, MIN-K% PROB achieved an average AUC of 0.72, outperforming previous methods by 7.4%. Larger models and longer texts improved detection accuracy. In copyright detection, it reached 0.88 AUC, identifying over 90% of copyrighted books. In dataset contamination scenarios, it outperformed baselines with a 12.2% increase in TPR@5%FPR. The method proved robust across different models and scenarios, demonstrating broad applicability.
Significance
This work addresses a critical challenge in AI transparency, privacy, and copyright protection. By enabling effective detection of pretraining data without requiring access to the data itself or training reference models, it offers a scalable, practical solution for industry and academia. It enhances trustworthiness of large models, supports legal compliance, and promotes ethical AI deployment, especially as models grow larger and more opaque.
Technical Contribution
The paper introduces a novel, reference-free membership inference attack method based on low-probability token detection, supported by a dynamic Wikipedia-based benchmark. It departs from traditional shadow model approaches, simplifying the detection process and reducing costs. Theoretical analysis links detection difficulty to model size and text length, validated through extensive experiments, establishing new standards for black-box data auditing.
Novelty
This is the first method to perform pretraining data detection without reliance on shadow or reference models, leveraging the statistical properties of low-probability tokens. Its dynamic benchmark and automatic data collection further distinguish it from prior static, model-dependent approaches, enabling scalable, real-time detection in evolving environments.
Limitations
- Detection accuracy may decrease with very short or paraphrased texts where low-probability tokens are less prominent.
- Computational costs remain high for extremely large models, especially in multi-model scenarios.
- Dependence on accurate timestamp data limits effectiveness if source information is unreliable or unavailable.
Future Work
Future research will focus on improving detection of heavily paraphrased or semantically altered texts, integrating semantic understanding. Extending the approach to multimodal data and real-time monitoring, as well as developing adaptive thresholds for diverse scenarios, are promising directions.
AI Executive Summary
As large language models (LLMs) like GPT-4 and LLaMA become integral to AI applications, concerns over training data transparency, privacy, and copyright infringement intensify. Despite their widespread deployment, the exact datasets used remain largely undisclosed, creating challenges for ethical and legal accountability. Traditional detection methods rely on training shadow models, which are computationally expensive and impractical at scale, especially given the enormous size of pretraining corpora.
This paper introduces MIN-K% PROB, a novel, reference-free detection algorithm that leverages the statistical property that unseen data typically contains a few outlier tokens with low probabilities under the model. By calculating the average negative log-likelihood of the lowest K% tokens in a text, the method effectively distinguishes between data seen during pretraining and novel data. The approach is simple, efficient, and applicable in black-box settings, making it suitable for real-world scenarios.
To evaluate the method, the authors built a dynamic benchmark called WIKIMIA, which uses Wikipedia event data timestamped before and after model training to create a reliable, evolving test set. Extensive experiments across multiple models, including LLaMA, GPT-Neo, and Pythia, demonstrated that MIN-K% PROB outperforms existing baselines by 7.4% in AUC, with detection accuracy improving as model size and text length increase.
The method proved effective in diverse applications: detecting copyrighted books in training data, identifying contaminated downstream datasets, and auditing privacy leaks. For instance, it achieved an AUC of 0.88 in copyright detection, revealing that over 90% of sampled books were likely included in GPT-3βs training corpus. In data contamination scenarios, it surpassed baselines with a 12.2% higher TPR at low FPR.
Overall, this work provides a scalable, practical tool for AI transparency and data governance, addressing a critical gap in current capabilities. Future directions include enhancing paraphrase detection, extending to multimodal data, and developing adaptive, real-time monitoring systems, paving the way for more trustworthy and ethically aligned AI systems.
Deep Dive
Abstract
Although large language models (LLMs) are widely deployed, the data used to train them is rarely disclosed. Given the incredible scale of this data, up to trillions of tokens, it is all but certain that it includes potentially problematic text such as copyrighted materials, personally identifiable information, and test data for widely reported reference benchmarks. However, we currently have no way to know which data of these types is included or in what proportions. In this paper, we study the pretraining data detection problem: given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model was trained on the provided text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that uses data created before and after model training to support gold truth detection. We also introduce a new detection method Min-K% Prob based on a simple hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data. Moreover, our experiments demonstrate that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous methods. We apply Min-K% Prob to three real-world scenarios, copyrighted book detection, contaminated downstream example detection and privacy auditing of machine unlearning, and find it a consistently effective solution.