Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
Unified protocol shows synthetic data can accelerate LLM pre-training; optimal mix around 30% synthetic improves efficiency 5-10x.
Key Findings
Methodology
This large-scale study trained over 1000 models (up to 3B parameters) using more than 100,000 GPU hours, comparing natural web data, rephrased synthetic texts, and textbook-style synthetic data. Using a unified scaling law framework, it analyzed how different synthetic data types, proportions, and generator model sizes influence training convergence, validation loss, and downstream task performance. The experiments employed a consistent transformer architecture based on Llama 3, with data mixtures ranging from pure natural to pure synthetic, across various model sizes and data budgets. Synthetic data was generated via pre-trained models like Mistral-7B, with prompts designed for high-quality and QA styles, and textbook-like content. The study fitted power-law models to predict validation loss as functions of data and model size, validating the effects of synthetic data on training efficiency and performance.
Key Results
- Pre-training solely on rephrased synthetic data does not outperform natural web data; however, mixing 1/3 synthetic with 2/3 natural data accelerates training by 5-10 times at large data budgets, reaching comparable validation loss.
- Pure textbook-style synthetic data results in higher validation loss across multiple domains, especially at small data budgets; optimal synthetic mixture ratios (~30%) depend on model size and data budget, with larger models being more sensitive.
- Increasing generator model size beyond 8B parameters does not necessarily produce better synthetic data for downstream training; 8B models outperform 70B models in synthetic data quality, challenging the assumption that bigger generators are always better.
Significance
This comprehensive empirical investigation clarifies the conditional benefits of synthetic data in large-scale pre-training, addressing longstanding concerns about model collapse and data efficiency. It provides practical guidelines for optimal synthetic data proportions, demonstrating that well-calibrated mixtures can significantly boost training speed without performance degradation. The findings have implications for reducing reliance on scarce high-quality natural data, enabling more efficient and accessible development of large language models, and informing future research on data generation strategies and scaling laws.
Technical Contribution
The study introduces a unified experimental framework combining multiple synthetic data generation paradigms with rigorous scaling law modeling. It empirically validates the power-law relationships between validation loss, data size, and model size, and proposes a 30% synthetic data ratio as a robust heuristic. The work also challenges the conventional wisdom that larger generator models always produce superior synthetic data, revealing nuanced effects of generator capacity on downstream performance. Additionally, it provides empirical evidence against the hypothesis of model collapse during synthetic data training, showing that appropriate mixture ratios do not degrade performance even at large scales.
Novelty
This research is the first to systematically evaluate multiple synthetic data generation methods and their interactions with natural data across extensive model and data scales within a unified framework. It combines empirical scaling law fitting with practical mixture ratio optimization, offering new insights into the conditional advantages of synthetic data. The work also critically examines the impact of generator model size, providing nuanced guidance that contradicts the simplistic ‘bigger is better’ paradigm prevalent in prior studies.
Limitations
- The experiments focus primarily on English text and models up to 3B parameters; applicability to multilingual or larger models remains to be validated.
- Synthetic data generation depends on pre-trained models, which may introduce biases or domain-specific limitations not fully explored here.
- Computational costs are substantial, and the optimal mixture ratios may vary with different architectures, training regimes, or downstream tasks, requiring further investigation.
Future Work
Future research should extend analysis to multilingual and multimodal data, explore adaptive mixture strategies, and incorporate reinforcement learning for dynamic data balancing. Investigating larger models (e.g., GPT-4 scale) and more diverse synthetic generation techniques will deepen understanding of synthetic data’s role. Developing automated tools for optimal mixture ratio tuning and bias mitigation in synthetic data will further enhance practical deployment, ultimately aiming for more efficient, scalable, and fair large language model training pipelines.
AI Executive Summary
The rapid growth of large language models (LLMs) has underscored the importance of training data quality and scale. However, high-quality natural data remains scarce and expensive, prompting interest in synthetic data as an alternative or supplement. This study systematically investigates the role of synthetic data in pre-training, leveraging a unified experimental protocol and scaling law models to analyze over 1000 models, up to 3B parameters, trained on datasets exceeding 200 billion tokens.
The core finding is that synthetic data alone does not accelerate training compared to natural web data, but when mixed at an optimal ratio (~30%), it can significantly boost training efficiency—by 5 to 10 times—without performance loss. Different synthetic data types, such as rephrased web texts and textbook-style content, exhibit distinct effects. Rephrased data, especially when combined with natural data, enhances convergence speed, whereas pure textbook synthetic data often results in higher validation loss, indicating limited utility in some contexts.
An important contribution is the validation of a power-law scaling law relating validation loss to data and model size, enabling accurate predictions of performance at larger scales. Surprisingly, increasing generator model size beyond 8B parameters does not necessarily improve synthetic data quality for downstream tasks, challenging the assumption that bigger models produce better synthetic data.
The research also addresses concerns about “model collapse,” finding that appropriate mixture ratios do not degrade performance, and that synthetic data can be effectively integrated into large-scale training pipelines. These insights provide practical guidelines for optimizing synthetic data use, potentially reducing dependence on scarce natural data and accelerating the development of more capable, efficient LLMs. Future work will explore adaptive mixture strategies, multilingual and multimodal applications, and larger models, aiming to refine synthetic data generation and deployment further.
Deep Analysis
Background
The evolution of large language models (LLMs) has been driven by increasing model sizes and vast datasets, exemplified by GPT, LLaMA, and PaLM series. Natural web data, such as CommonCrawl, has been the primary source, but its limited high-quality content constrains further scaling. Synthetic data, generated via models like GPT or Mistral, offers a promising avenue to augment or replace natural data, especially for tasks like instruction tuning and alignment. Prior studies demonstrated benefits in downstream tasks but lacked systematic analysis during foundational pre-training. Theoretical debates about data diversity, quality, and risks like model collapse remain unresolved, necessitating comprehensive empirical investigations to guide effective synthetic data utilization.
Core Problem
The key challenge is determining whether synthetic data can reliably accelerate large-scale pre-training without compromising performance. Existing evidence is fragmented, with inconsistent findings on the impact of synthetic data types, proportions, and generator capabilities. The risk of “model collapse”—performance degradation due to recursive synthetic data training—raises concerns about long-term stability. Additionally, understanding the optimal mixture ratios, the influence of generator size, and the scalability of benefits across model sizes and data budgets remains elusive. Addressing these issues is critical for making synthetic data a practical component of scalable, efficient pre-training pipelines.
Innovation
This work introduces a comprehensive, empirical framework combining large-scale experiments with rigorous scaling law modeling. It systematically compares multiple synthetic data generation paradigms—rephrased web texts and textbook-style content—and their mixtures with natural web data across diverse model sizes and data budgets. The study validates power-law relationships between validation loss, data volume, and model size, establishing a practical “30% synthetic data” rule. It also challenges the assumption that larger generator models always produce better synthetic data, revealing nuanced effects of generator capacity. Importantly, it empirically demonstrates that appropriate synthetic data mixtures do not induce performance collapse, providing a new understanding of synthetic data’s role in scalable pre-training.
Methodology
- �� Design large-scale pre-training experiments with models from 100M to 3B parameters, trained on datasets up to 200B tokens.
- �� Generate synthetic data using pre-trained models (e.g., Mistral-7B) with prompts for high-quality and QA styles, as well as textbook-like content.
- �� Construct data mixtures with varying synthetic proportions (0%, 33%, 67%, 100%) and natural data, ensuring multiple runs for statistical robustness.
- �� Fit power-law models to relate validation loss to data size and model size, validating predictive accuracy.
- �� Analyze the effects of synthetic data type, mixture ratio, and generator capacity on training speed, validation loss, and downstream task performance.
- �� Evaluate the irreducible loss and potential for model collapse across different configurations, assessing stability and performance degradation risks.
Experiments
- �� Use CommonCrawl as the natural baseline, processed via RedPajama-v2 pipeline.
- �� Generate synthetic datasets with Mistral-Instruct-7B, covering rephrased (HQ, QA) and textbook styles.
- �� Mix synthetic and natural data at multiple ratios, train models from 100M to 3B parameters, with data budgets up to 200B tokens.
- �� Employ consistent training hyperparameters (learning rate schedule, batch size, context length).
- �� Fit scaling laws to validation loss data, validate predictions at larger scales (up to 400B parameters), and analyze the impact of synthetic data proportions.
- �� Conduct ablation studies on generator model size and synthetic mixture ratios to identify optimal configurations.
Results
- �� Synthetic data alone does not outperform natural data, but mixing 30% synthetic with natural data significantly accelerates training, achieving similar validation loss 5-10x faster.
- �� Pure textbook synthetic data performs worse than natural data, with optimal synthetic ratios around 30%; exceeding this ratio degrades performance.
- �� Increasing generator size beyond 8B parameters does not improve synthetic data quality for downstream tasks; 8B models outperform larger ones.
- �� Power-law models accurately predict validation loss across data and model scales, validating the scalability of synthetic data benefits.
Applications
- �� Immediate: Enhance training efficiency for resource-constrained organizations by incorporating optimized synthetic data mixtures.
- �� Long-term: Enable scalable, cost-effective AI development with reduced dependence on scarce high-quality natural datasets, facilitating multilingual and multimodal model training.
Limitations & Outlook
- �� Focused on English text and models up to 3B parameters; applicability to larger models and other languages needs validation.
- �� Synthetic data generation depends on pre-trained models, which may embed biases.
- �� High computational costs limit rapid experimentation; optimal ratios may vary across architectures and tasks.
Plain Language Accessible to non-experts
想象你在准备一场盛大的派对，食材就像训练模型用的数据。自然网页文本就像你从市场买来的新鲜食材，但有时候这些食材不够多，不能满足所有的需求。于是，你请厨师（合成模型）帮忙，把已有的食材变成不同的菜肴（合成数据）。如果只用厨师做的菜，可能味道不够丰富，但如果你把一部分新鲜食材和厨师做的菜混在一起，就能做出又快又好吃的饭。不同厨师（生成模型）能力不同，能力强的厨师能做出更好吃的菜，但不一定比中等厨师更适合所有菜肴。这样，食材和厨师的比例就像训练中的数据比例和生成模型的能力，合理搭配可以让你做出最棒的饭，既省时间又保证质量。
Abstract
Training data plays a crucial role in Large Language Models (LLM) scaling, yet high quality data is of limited supply. Synthetic data techniques offer a potential path toward sidestepping these limitations. We conduct a large-scale empirical investigation (>1000 LLMs with >100k GPU hours) using a unified protocol and scaling laws, comparing natural web data, diverse synthetic types (rephrased text, generated textbooks), and mixtures of natural and synthetic data. Specifically, we found pre-training on rephrased synthetic data \textit{alone} is not faster than pre-training on natural web texts; while pre-training on 1/3 rephrased synthetic data mixed with 2/3 natural web texts can speed up 5-10x (to reach the same validation loss) at larger data budgets. Pre-training on textbook-style synthetic data \textit{alone} results in notably higher loss on many downstream domains especially at small data budgets. "Good" ratios of synthetic data in training data mixtures depend on the model size and data budget, empirically converging to ~30% for rephrased synthetic data. Larger generator models do not necessarily yield better pre-training data than ~8B-param models. These results contribute mixed evidence on "model collapse" during large-scale single-round (n=1) model training on synthetic data--training on rephrased synthetic data shows no degradation in performance in foreseeable scales whereas training on mixtures of textbook-style pure-generated synthetic data shows patterns predicted by "model collapse". Our work demystifies synthetic data in pre-training, validates its conditional benefits, and offers practical guidance.