Explaining Neural Scaling Laws

Recent advances in deep learning have shown that model performance scales predictably with parameters and data, often following power laws. Early empirical studies, such as those by Kaplan et al. (2020), identified these scaling laws but lacked a unified theoretical explanation. Subsequent work explored kernel methods and data geometry, revealing partial insights into the mechanisms. However, a comprehensive framework that classifies different scaling regimes and links them to data and model properties was missing. As models grow larger, understanding the microscopic origins of these laws becomes crucial for efficient scaling and resource allocation. This paper builds on these foundations, integrating kernel spectral analysis and data manifold theory to explain the observed power laws systematically.

Despite empirical evidence of power-law scaling, the fundamental causes—whether driven by variance reduction, data resolution, or other factors—remain unclear. Existing theories often focus on specific regimes, lacking a unified view. The key challenge is to identify the mechanisms governing performance improvements across different scales and to establish explicit relationships between model size, data complexity, and error decay. Understanding these relationships is vital for optimizing large-scale training, reducing costs, and improving generalization. Moreover, clarifying the duality between data and model scaling can lead to more flexible and resource-efficient strategies, especially as models approach practical limits.

This work introduces a systematic classification of four scaling regimes: variance-limited and resolution-limited, each with respect to data and model size. It employs kernel spectral decay to quantify the resolution limits, linking eigenvalue decay rates to data manifold dimension. The duality between model and data scaling is a novel conceptual insight, suggesting that performance gains can be achieved equivalently by increasing data or model complexity. The authors rigorously derive these relationships within random feature models and validate them empirically on deep neural networks, bridging the gap between theory and practice. This comprehensive framework advances understanding of the microscopic mechanisms behind neural scaling laws.

• �� Develop random teacher-student models with feature covariance matrices and kernel functions. • Derive variance-limited exponents by analyzing fluctuations in finite data and width limits, employing concentration inequalities. • Use kernel spectral decay to quantify resolution limits, relating eigenvalue decay rates to data manifold dimension. • Establish duality by mathematically linking model size and data size effects via spectral properties. • Validate predictions through extensive experiments on MNIST, CIFAR-10, CIFAR-100, varying architecture parameters, data augmentation, and noise. • Measure error decay and extract exponents, comparing with theoretical formulas. • Analyze the influence of data distribution and task modifications on the exponents, confirming the mechanisms' robustness.

Experiments involve training random feature models and deep neural networks on standard datasets, measuring test error across varying model widths and dataset sizes. The models include CNNs, ResNets, and pretrained transformers, with hyperparameters tuned for convergence. Data is manipulated via downsampling, noise addition, and class super-classing to test the impact on scaling exponents. Kernel spectra are computed to verify the spectral decay hypothesis. Results show consistent power-law behavior matching theoretical predictions, with exponents varying systematically with data complexity and architecture. These experiments confirm the duality and the role of kernel properties in performance scaling.

Empirical data confirms four regimes: variance-limited with exponent 1, and resolution-limited with exponents matching kernel spectral decay αK. Deep networks exhibit these regimes across datasets, with exponents sensitive to data distribution and architecture. The duality between data and model scaling is validated, showing that increasing either can achieve similar error reductions. Kernel spectra follow power-law decay, directly influencing the resolution-limited exponents. Results demonstrate that understanding kernel properties and data geometry enables more efficient scaling strategies, reducing training costs while maintaining accuracy.

The framework guides the design of scalable architectures, optimal data collection, and training strategies. It informs hyperparameter tuning by predicting error decay rates, aiding resource allocation. In industry, it can optimize transfer learning, model compression, and active data sampling, reducing costs. The theory also provides insights for developing adaptive training protocols that leverage data-model duality, improving efficiency in large-scale AI systems.

The analysis assumes smooth kernels and data manifolds, which may not hold in highly nonlinear, real-world scenarios. The models focus on linearized regimes, limiting applicability to complex deep architectures. Experimental validation is primarily on standard datasets, requiring further testing in diverse, real-world environments. The influence of training dynamics, regularization, and non-stationary data remains to be studied. Extending the theory to account for non-smoothness and nonlinearity is an important future step.