InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
Introduces InfoLaw, a data-aware scaling law using information theory to predict large language model performance with quality-weighted data and repetition, achieving 0.15% loss prediction error.
Key Findings
Methodology
This paper develops the InfoLaw framework, modeling pretraining as an information accumulation process. It incorporates data quality via an exponential decay function, models the impact of repetition, and relates total information to model loss through a power-law. Using LayerMix sampling, datasets with varied quality and repetition levels are constructed. Multiple models (252M-7B) are trained across configurations, and performance data are collected. The information metric, combining data quality density and repetition effects, predicts unseen data and larger-scale results with a mean error of 0.15%. The framework enables efficient data recipe selection and robust extrapolation across scales and data mixtures.
Key Results
- InfoLaw accurately predicts loss within 0.15% error on models up to 7B parameters trained on 425B tokens, outperforming traditional scaling laws especially under data repetition. It generalizes well to unseen data recipes and higher overtraining ratios, guiding optimal data mixture choices. The information metric collapses diverse configurations onto a unified power-law curve, facilitating reliable performance forecasting.
- Compared to classical power-law scaling, which overestimates performance under repeated data, InfoLaw accounts for diminishing returns via the information decay model. Experimental results show that InfoLaw maintains prediction accuracy across different data quality distributions and model sizes, supporting its use for training optimization. The fitted parameters reveal a non-linear relationship between model capacity and information gain, validating the theoretical model.
- In practical terms, InfoLaw enables researchers to simulate and optimize data configurations before training, saving computational resources. It provides a principled approach to balancing data quality and quantity, especially in data-limited regimes. The framework's robustness across scales and data recipes demonstrates its potential to standardize large model training strategies, reducing trial-and-error and improving resource efficiency.
Significance
This work addresses a fundamental challenge in large-scale model training: how to predict and optimize performance considering data heterogeneity and repetition. By introducing an information-centric scaling law, it overcomes the limitations of traditional laws that ignore data quality effects. The framework offers a systematic way to select data mixtures and predict outcomes, significantly advancing the theoretical understanding of model scaling. Its practical implications include reducing training costs, improving model robustness, and guiding data curation strategies. The approach bridges the gap between empirical observations and theoretical modeling, setting a new standard for large model training optimization.
Technical Contribution
The paper introduces a novel information-theoretic scaling law (InfoLaw) that models the training process as an accumulation of information, incorporating data quality via a density function and data repetition via an exponential decay. It combines these into a unified power-law relationship with model loss, validated across multiple scales and data configurations. The framework leverages a new sampling strategy (LayerMix) and fits key parameters (quality density and decay rate) through extensive experiments, enabling accurate extrapolation. This approach advances the theoretical foundation of scaling laws by explicitly modeling data heterogeneity and repetition effects, providing a practical tool for training large models efficiently.
Novelty
This is the first work to explicitly incorporate data quality distribution and repetition effects into a unified information-based scaling law for large language models. Unlike prior empirical laws focusing solely on model size or data volume, InfoLaw models the diminishing returns of repeated high-quality data through an exponential decay mechanism, enabling reliable prediction across unseen data recipes and larger scales. Its use of an information metric to collapse diverse configurations onto a single power-law curve represents a significant conceptual and practical innovation, bridging theory and real-world training scenarios.
Limitations
- The exponential decay assumption may oversimplify complex data interactions, especially in extreme cases of high repetition or very low-quality data. Parameter fitting relies on specific datasets, limiting generalization to radically different data distributions. Computational costs for extensive training and fitting remain high. Further work is needed to incorporate dynamic data environments and multi-modal data sources, enhancing robustness and applicability in diverse real-world settings.
Future Work
Future directions include extending the InfoLaw framework to multi-modal data, dynamic data environments, and reinforcement learning-based data selection. Improving the decay model to capture more complex data interactions and scaling behaviors is also planned. Additionally, integrating the framework into automated training pipelines could enable real-time data optimization, reducing costs and improving performance. Exploring theoretical bounds and robustness under different data regimes will further solidify its utility for large-scale AI systems.
AI Executive Summary
The rapid growth of large language models (LLMs) has transformed natural language processing, yet optimizing their training remains a challenge. Traditional scaling laws, which relate model performance to size and data volume, fall short when data quality varies or when datasets contain significant repetition. These limitations hinder efficient resource allocation and data curation strategies, especially as models grow beyond hundreds of billions of parameters.
This paper introduces the innovative InfoLaw framework, which models pretraining as an information accumulation process. By quantifying data quality through an information density function and modeling the diminishing returns caused by data repetition via exponential decay, the authors establish a unified power-law relationship between total information and model loss. This approach captures the complex interplay between data heterogeneity, repetition, and model scale, providing a more accurate predictive tool.
Using extensive experiments with datasets derived from Common Crawl, the authors trained models ranging from 252 million to 7 billion parameters across various data configurations. The results demonstrate that InfoLaw predicts model loss with a mean error of only 0.15%, significantly outperforming classical scaling laws, especially under high-repetition scenarios. The information metric effectively collapses diverse data recipes and model sizes onto a single predictive curve, enabling precise performance forecasting and optimal data recipe selection.
Beyond theoretical insights, this work offers practical benefits. It allows researchers to simulate training outcomes, optimize data mixtures, and reduce computational costs before actual training. The framework's robustness across unseen data recipes and larger scales paves the way for more efficient, resource-aware large model training strategies. Looking ahead, the authors plan to extend the model to multi-modal data, incorporate dynamic data environments, and develop automated data optimization pipelines, promising a new era of intelligent large-scale AI systems.
Deep Dive
Abstract
Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.