Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation
DataEvolve employs evolutionary algorithms and sample evaluation to autonomously optimize pretraining data strategies, boosting model performance.
Key Findings
Methodology
DataEvolve adopts a closed-loop evolutionary framework comprising data quality detection, strategy generation, sample execution, and evaluation. It maintains an experience pool for issues and a strategy pool for performance tracking, enabling multi-generation optimization. Strategies are generated via diagnostic feedback and mutation, guided by specific algorithms combining genetic search and sample-based fitness. Experiments on 8 categories with 30 generations each produce Darwin-CC, a 504B-token dataset, outperforming raw data by 3.96 points on average across 18 benchmarks. The strategies tend toward noise removal and format normalization, reflecting L4 (Generative Refinement) principles.
Key Results
- Training 3B parameter models on 500B tokens, Darwin-CC improves average benchmark scores to 44.13, surpassing raw data by 3.96 points, with notable gains on knowledge-intensive tasks like MMLU (+18.64 points).
- Compared to manually designed strategies, evolved strategies outperform in 18 benchmarks, confirming the effectiveness of automated optimization. Ablation studies show non-evolved strategies only gain 1 point, highlighting evolution's importance.
- Analysis reveals strategies converge on cleaning-focused approaches, preserving domain-specific features, validating the method's robustness and convergence.
Significance
This work addresses the scalability bottleneck in manual data curation for large-scale pretraining, transforming it into an automated, scalable process. It significantly enhances data quality and model performance, setting a new paradigm for AI data management. By automating strategy design, it reduces reliance on expert knowledge and accelerates the development cycle, enabling more efficient and effective large-scale training pipelines.
Technical Contribution
The paper introduces a novel evolutionary framework combining genetic algorithms with sample-based fitness evaluation, supported by knowledge pools for cross-generation learning. It shifts from manual rule-based filtering to automated, adaptive strategy optimization, providing theoretical guarantees of convergence and practical scalability for multi-category data curation. The approach enables rapid, cost-effective refinement of data strategies at pretraining scale.
Novelty
This is the first application of evolutionary algorithms for automatic strategy optimization in large-scale pretraining data curation. Unlike prior filtering or manual tuning, DataEvolve enables autonomous, category-specific strategy evolution, demonstrating superior performance and scalability. Its convergence on cleaning-focused approaches underscores the potential for simple yet effective automated solutions.
Limitations
- The reliance on large language models for sample evaluation incurs high computational costs, limiting real-time or resource-constrained applications.
- Current strategies mainly focus on noise removal and formatting, with less emphasis on content diversity and knowledge enrichment.
- Generalization to unseen categories or highly specialized domains remains untested, requiring further validation and adaptation.
Future Work
Future research will explore multi-objective optimization, integrating content richness and semantic preservation. Combining reinforcement learning and meta-learning could enhance strategy adaptability. Extending the framework to multimodal data and real-time applications will broaden its impact, aiming for fully autonomous, scalable data curation systems.
AI Executive Summary
Large-scale pretraining of language models hinges critically on the quality of training data. Traditional strategies rely heavily on manual filtering and heuristic rules, which become infeasible as data volume and diversity grow exponentially. Addressing this challenge, the authors propose DataEvolve, an innovative framework that leverages evolutionary algorithms and sample-based evaluation to autonomously optimize data curation strategies across hundreds of categories.
The core of DataEvolve is a closed-loop system comprising four components: data quality detection, strategy generation, sample execution, and performance evaluation. This cycle iteratively refines strategies by mutating and selecting the best performers, guided by diagnostic feedback stored in experience and strategy pools. The process enables the automatic discovery of category-specific cleaning rules, focusing on noise removal and format normalization, which are validated through extensive experiments.
Applied to the Nemotron-CC corpus, covering 8 categories with 672 billion tokens, the framework evolved strategies over 30 generations per category. The resulting Darwin-CC dataset, containing 504 billion tokens, demonstrated significant improvements in downstream model performance. Models trained on Darwin-CC achieved an average score of 44.13 across 18 benchmarks, outperforming models trained on raw data (+3.96 points) and comparable or superior to existing curated datasets like DCLM and Ultra-FineWeb. Notably, gains were especially pronounced on knowledge-intensive tasks such as MMLU (+18.64 points), confirming the effectiveness of the evolved strategies.
Analysis of the strategies revealed a convergence on cleaning-focused approaches, emphasizing targeted noise removal and domain-aware preservation. This aligns with the L4 principles from Part I, indicating that simple, well-evolved cleaning rules can unlock substantial data value without complex transformations. Ablation studies confirmed the necessity of iterative evolution, as non-evolved strategies lagged behind by nearly 3 points.
Overall, this work demonstrates that automated, evolutionary strategy design is both feasible and essential at pretraining scale. It paves the way for scalable, adaptive data curation systems that can keep pace with the rapid growth of AI models, reducing reliance on manual expertise and enabling more efficient training pipelines. Future directions include multi-objective optimization, multimodal extension, and real-time deployment, promising a new era of intelligent data management for AI development.
Deep Analysis
Background
近年来，随着大规模预训练模型的兴起，数据质量成为限制模型性能的关键因素。早期工作如WebText、C4采用简单过滤策略，但面对多样化内容和类别时效果有限。近年来，内容重构和知识增强方法逐步兴起，但大多依赖人工规则，难以扩展到数百类别。传统方法在规模和适应性方面存在瓶颈，难以满足未来模型对高质量、多样化数据的需求。自动化策略优化成为解决方案的关键，但现有研究多集中在单一类别或有限内容类型，缺乏系统性框架。本文提出的DataEvolve结合遗传算法和样本评估，为多类别数据策略自动优化提供了理论基础和实践路径。
Core Problem
当前预训练数据策略设计主要依赖人工经验，难以应对数百类别的多样性和规模。手工调优耗时耗力，难以实现大规模自动化。如何在保证策略效果的同时，降低人力成本，提升适应性，成为核心难题。此外，缺乏有效的自动化机制探索和优化不同类别的特定策略，导致数据质量参差不齐，影响模型性能。解决这一问题，需开发可扩展、自动化、智能的策略优化框架，满足未来大规模预训练的需求。
Innovation
本研究的创新点在于引入基于遗传算法的策略演化机制，结合样本评估和知识池管理，实现多类别策略的自动优化。区别于传统过滤和手工调优，DataEvolve通过诊断反馈引导变异，避免盲目搜索，显著降低成本。其核心在于：
- �� 设计闭环演化流程，包括质量检测、策略生成、样本执行、效果评估；
- �� 利用样本评估替代全模型训练，提升效率；
- �� 通过知识池积累问题与经验，实现跨代学习与优化。这些创新使得自动化策略设计成为可能，并在大规模预训练中展现出优越性能。
Methodology
- �� 数据质量检测：利用GPT-4-mini分析样本，识别类别特定的噪声和格式问题；• 策略生成：结合经验池信息，利用GPT-4-mini合成初始策略，后续通过遗传变异优化；• 样本执行：用GPT-OSS-120B在样本数据上应用策略，生成清洗后文本；• 评估：由gpt-4o对样本对进行打分和诊断，获得策略效果指标；• 反馈机制：将问题与效果存入经验池和策略池，指导下一轮变异；• 迭代优化：重复上述步骤30轮，逐步演化出最优策略，最终应用于全数据集。
Experiments
采用Nemotron-CC数据集，涵盖8个类别，总计672B tokens。每类别进行30轮演化，利用GPT-4-mini分析样本，GPT-4-120B执行策略，gpt-4o评估。训练3B参数模型，使用500B tokens，比较原始数据、低效策略和演化策略的模型性能。评估指标包括18项基准测试的平均得分，特别关注知识密集任务的提升。通过ablation验证演化的重要性，分析策略收敛性和效果差异。
Results
演化策略显著优于手工和非演化策略，模型平均得分从40.17提升至44.13，提升3.96点。知识任务如MMLU提升18.64点，验证了策略在保持事实信息方面的有效性。ablation显示未演化策略仅提升1点，验证演化的必要性。分析策略发现趋向于噪声去除和格式标准化，体现L4原则，验证了演化过程的有效性和收敛性。
Applications
该方法可广泛应用于大规模预训练数据的自动筛选与清理，特别适合多类别、多内容类型的场景。企业和研究机构可利用此框架提升数据质量，减少人工成本，加快模型训练周期。未来，结合内容丰富性和知识增强，将推动AI在教育、医疗、科研等领域的应用革新。
Limitations & Outlook
目前方法依赖大规模LLM样本评估，计算成本较高，难以实时应用。策略主要集中在噪声过滤和格式标准化，内容丰富性和知识深度仍需提升。对极端类别或特殊任务的适应性有限，未来需结合多目标优化和强化学习，增强泛化能力。
Plain Language Accessible to non-experts
想象你在整理一个巨大的图书馆，里面有各种各样的书籍。有些书可能夹杂着错误的内容、格式不一致，甚至重复。你希望把这些书整理得更干净、更有用，但每次手动整理太费时间，也难以应对如此庞大的书库。于是，你设计了一套自动化的“清理机器人”。这个机器人可以先扫描一部分书，找出常见的问题，比如错别字、格式不统一、重复内容。然后，它会尝试不同的清理方法，比如删除多余的广告、规范排版、保留专业术语。每次清理后，它会检查效果，看看哪些方法最有效。通过不断试错和学习，机器人逐渐掌握了最适合这个图书馆的整理策略。这就像本文中的DataEvolve，用AI让数据“自己学习”变得更干净、更有价值。最终，整理好的书库让读者更容易找到所需内容，也让图书馆管理变得更高效。
ELI14 Explained like you're 14
想象你有一个超级大的学校，每天都要整理很多学生的作业。有些作业乱七八糟，有拼写错误，有的格式不一样。你希望让所有作业都变得整整齐齐、没有错误，但手动检查太慢了。于是，你让一个聪明的机器人帮忙。这个机器人可以先看一部分作业，找出常见的问题，比如错字、格式不一致、重复内容。然后，它会试着用不同的方法来改正，比如删除多余的内容、统一排版、保留重要的内容。每次改完后，它会检查效果，看哪些方法效果最好。通过不断试错和学习，机器人逐渐掌握了最有效的整理技巧。这样，所有的作业都变得更整洁，老师也省了很多时间。这就像论文里的DataEvolve，让AI自己学习怎么整理和清理大量数据。最终，整理好的数据可以帮助训练出更聪明、更可靠的模型，就像学生的作业变得更棒一样。
Abstract
Data Darwinism (Part I) established a ten-level hierarchy for data processing, showing that stronger processing can unlock greater data value. However, that work relied on manually designed strategies for a single category. Modern pretraining corpora comprise hundreds of heterogeneous categories spanning domains and content types, each demanding specialized treatment. At this scale, manual strategy design becomes prohibitive. This raises a key question: can strategies evolve in an automated way? We introduce DataEvolve, a framework that enables strategies to evolve through iterative optimization rather than manual design. For each data category, DataEvolve operates in a closed evolutionary loop: it identifies quality issues, generates candidate strategies, executes them on sampled data, evaluates results, and refines approaches across generations. The process accumulates knowledge through an experience pool of discovered issues and a strategy pool tracking performance across iterations. Applied to 8 categories spanning 672B tokens from Nemotron-CC, DataEvolve produces Darwin-CC, a 504B-token dataset with strategies evolved through 30 iterations per category. Training 3B models on 500B tokens, Darwin-CC outperforms raw data (+3.96 points) and achieves a 44.13 average score across 18 benchmarks, surpassing DCLM, Ultra-FineWeb, and FineWeb-Edu, with strong gains on knowledge-intensive tasks such as MMLU. Analysis shows evolved strategies converge on cleaning-focused approaches: targeted noise removal and format normalization with domain-aware preservation, echoing the L4 (Generative Refinement) principles from Part I. Ablation studies confirm iterative evolution is essential: optimized strategies outperform suboptimal ones by 2.93 points, establishing evolutionary strategy design as feasible and necessary for pretraining-scale data curation.