Internal Data Repetition Destroys Language Models
Using a fitted no-repetition scaling law, quantifies systematic damage caused by repeated data, revealing an intermediate repeat count causes maximum loss, with damage scaling as a power law with model size.
Key Findings
Methodology
This study leverages Chinchilla’s compute-efficient scaling laws, integrating precise control over repeated data (fixed 10% token fraction) across models from 34M to 344M parameters. It measures validation loss as a function of repeat count R, fitting a no-repetition loss curve to compute the Compute-Equivalent Gain (CEG) and Loss (CEL). A simple statistical linear regression model with verbatim duplicates explains the non-monotonic loss peak, emphasizing the statistical tradeoff between memorization and generalization. Experiments involve training models with varying overtraining multipliers (OT) and repeat counts, analyzing how intermediate R values maximize validation loss, and scaling laws predict the location of these peaks across model sizes.
Key Results
- On a 344M parameter Qwen3 model, the most damaging repeat count R ≈ 155 results in a CEL of 0.33, equivalent to training with only 67% of the FLOPs needed for no repetition, indicating significant compute waste.
- The peak repeat count Rpeak scales approximately as N^−0.96, showing larger models reach their worst damage at fewer repeats, with damage increasing as a power law with model size.
- The statistical model reproduces the non-monotonic loss peak, attributing it to a tradeoff between memorization and generalization, independent of specific model architecture, highlighting a fundamental statistical effect.
Significance
This work advances understanding of data duplication’s systemic impact on large-scale language models by quantifying damage in compute units rather than parameters. It provides a theoretical framework to optimize data sampling strategies, reducing redundant computation and improving training efficiency. The findings inform best practices for dataset curation, especially as models scale, and suggest that controlling repetition can significantly enhance model performance and resource utilization, addressing a long-standing challenge in scalable NLP training.
Technical Contribution
The paper introduces a novel metric, CEG and CEL, grounded in the compute-efficient scaling law, enabling cross-model comparisons of repetition damage. It combines empirical experiments with a simple statistical linear regression model to explain the non-monotonic loss behavior. The power-law relationships derived for peak damage location and pool size provide predictive tools for identifying harmful repetition regimes, contributing both theoretical insight and practical guidance for training large models efficiently.
Novelty
This is the first comprehensive quantification of repetition damage in terms of compute units, utilizing the Chinchilla scaling law. It uniquely combines empirical data with a statistical model to explain non-monotonic loss peaks, offering a new perspective on the impact of data duplication. The approach moves beyond traditional parameter-based metrics, providing a scalable, predictive framework applicable across different model sizes and training regimes, marking a significant advancement in understanding data efficiency in language modeling.
Limitations
- Experiments are limited to the FineWeb-Edu-Dedup dataset and Qwen3-style models, requiring validation on other datasets and architectures for broader applicability.
- The statistical model simplifies complex training dynamics, not accounting for attention mechanisms, optimizer effects, or semantic duplication types.
- Scaling laws are fitted within a limited parameter range; extrapolation to very large models remains uncertain, necessitating further empirical validation.
Future Work
Future research will explore diverse duplication types, including paraphrasing and semantic similarity, and extend the statistical framework to more complex architectures. Developing adaptive training strategies that dynamically detect and mitigate harmful repetition could further improve efficiency. Additionally, integrating automatic data filtering based on repetition structure promises to optimize resource use in large-scale training, ultimately enabling more effective and resource-conscious language models.
AI Executive Summary
As large-scale language models continue to grow, the challenge of training data quality and efficiency becomes increasingly critical. Despite efforts to eliminate duplicates, web data inevitably contains near-duplicate documents, templates, and redundant content, which can significantly impair model performance. Traditional metrics, focusing on parameter count, fail to capture the true computational cost of such repetition. This study leverages the modern Chinchilla scaling laws, which relate model size, data, and compute, to quantify the damage caused by exact document repetition in terms of compute units.
By conducting controlled experiments across models from 34 million to 344 million parameters, the authors reveal a non-monotonic relationship between repetition count and validation loss. Specifically, they find that repeating a moderately sized subset of data a moderate number of times causes the greatest performance degradation, with the loss peaking at an intermediate repeat count R. The damage then diminishes as repetition increases further, a phenomenon explained by a simple statistical linear regression model that captures the tradeoff between memorization and generalization.
The key insight is that the most damaging repetition structure scales with model size as a power law, Rpeak ∝ N^−0.96, allowing practitioners to predict harmful repetition regimes. Quantitatively, for a 344M parameter model, the worst-case damage corresponds to a loss equivalent to training with only 67% of the FLOPs needed for a no-repetition scenario, indicating substantial compute wastage. These findings highlight the importance of controlling repetition structure during pretraining, especially as models scale, to improve efficiency and performance.
Overall, this work provides a rigorous, scalable framework to quantify and mitigate the impact of data duplication, guiding future strategies for dataset curation and training optimization in large language models. It emphasizes that avoiding intermediate repetition regimes can significantly reduce wasted compute and enhance model generalization, paving the way for more resource-efficient NLP systems.
Deep Dive
Glossary
Scaling Law (规模定律)
描述模型性能随参数规模和计算量变化的数学关系，帮助预测不同规模下的表现。
用于分析重复数据对模型性能的影响。
Compute-Equivalent Loss (计算等价损失)
衡量重复数据带来的性能损失，转换为等效的计算成本。
用以量化重复数据的实际影响。
Repetition Structure (重复结构)
指重复样本的数量和重复次数的配置方式，影响模型的学习效果。
在实验中控制以研究损害机制。
No-repetition Scaling Law (无重复规模定律)
在没有重复数据的理想模型中，性能与计算量的关系。
作为衡量重复损害的基准。
Statistical Tradeoff (统计权衡)
模型在记忆重复样本与泛化能力之间的平衡，影响性能表现。
解释非单调损失峰的根源。
Open Questions Unanswered questions from this research
- 1 如何在实际大规模训练中自动检测和调整重复结构，减少计算浪费仍未解决。
- 2 语义重述和近似重复对模型损害的影响机制尚不明确，需进一步研究。
Abstract
Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition. Earlier controlled studies predated Chinchilla-style scaling laws and could only measure the cost of repetition indirectly. We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss. We show that under this modernized paradigm, repetition damage is systematic in three ways. First, holding compute allocated to repeated data constant, eval loss peaks at an intermediate repeat count $\Rep$; repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. Second, the location of this peak is well-fit by a power law in model size; this scaling law reveals that the most damaging number of repeated data grows more quickly than compute. Finally, when repeated documents consume 10\% of the FLOPs budget in a controlled exact-document repetition setting, the compute-equivalent loss can be large: on FineWeb-Edu-Dedup, the most damaging repeat count for a Qwen3-style 344M-parameter model at $\OT=1$ matches the loss of a no-repetition run using 67% of the FLOPs. We demonstrate that these phenomena are not language-model-specific, and can be analytically understood in a simple statistical model: a misspecified linear regression with verbatim duplicates reproduces the same qualitative loss peak, quantifying how such peaks can arise from a statistical tradeoff between memorization and generalization. Our findings add precision to the study of duplication in language models, allowing practitioners to quantify the wasted compute incurred by the presence and repeat structure of duplicates in pretraining corpora.