Scaling description of generalization with number of parameters in deep learning
Using Neural Tangent Kernel (NTK), the study reveals that output fluctuations scale as N^{-1/4}, leading to a generalization error decay of N^{-1/2} with a critical point N^{*}.
Key Findings
Methodology
This work employs the NTK framework to analyze the finite-size fluctuations of neural network outputs at initialization. It demonstrates that these fluctuations follow an N^{-1/4} scaling law, influencing the classification generalization error, which decays as N^{-1/2} towards a plateau. The analysis combines theoretical derivations with empirical validation on MNIST and CIFAR datasets, revealing a phase transition at N^{*} where the network norm diverges, explaining the observed cusp in test error. The approach links kernel methods with deep network training dynamics, providing a unified understanding of overparameterized regimes.
Key Results
- The output function fluctuations scale as N^{-1/4}, resulting in a generalization error that decreases as N^{-1/2} and approaches a plateau. Empirical results on MNIST and CIFAR confirm the power-law decay and the existence of a critical N^{*}. Ensemble averaging of several networks just beyond N^{*} significantly reduces error, outperforming single large models.
- Near N^{*}, the network norm diverges, causing a cusp in test error consistent with the theoretical predictions. The error decay rate matches the predicted N^{-1/2} exponent, validated by extensive experiments. The variance reduction through ensemble methods is quantitatively explained by the fluctuation analysis.
- Experiments demonstrate that the test error plateau scales inversely with network size, and ensemble strategies with intermediate N improve robustness and accuracy, especially around the phase transition point.
Significance
This research advances the fundamental understanding of how neural network generalization scales with parameters, bridging the gap between empirical observations and theoretical models. It clarifies why overparameterized networks generalize well despite their capacity to fit random data, attributing this to the finite-size fluctuations governed by NTK dynamics. The findings inform optimal network sizing and ensemble strategies, guiding practical model design and training in real-world applications such as image classification and natural language processing. The identification of a critical point N^{*} and the divergence of the network norm at this threshold provide new insights into the stability and capacity limits of deep models, with implications for avoiding overfitting and improving robustness.
Technical Contribution
The paper introduces a rigorous analysis of finite-size fluctuations in neural networks using NTK, deriving the N^{-1/4} scaling law for output variance. It establishes a power-law decay of generalization error as N^{-1/2} and characterizes the phase transition at N^{*} where the network norm diverges. The work combines probabilistic methods, kernel theory, and empirical validation, offering a novel theoretical framework that connects the training dynamics, spectral properties of NTK, and generalization performance. This approach extends beyond previous mean-field or VC-based analyses, providing precise quantitative predictions for large but finite networks.
Novelty
This study is the first to rigorously link the finite-size fluctuations of neural network outputs to the power-law decay of generalization error, explicitly deriving the N^{-1/4} fluctuation law and the N^{*} divergence phenomenon. Unlike prior work focusing on asymptotic limits or simplified models, it combines kernel theory with detailed fluctuation analysis, offering a comprehensive picture of the overparameterized regime. The identification of the critical point N^{*} as a phase transition with norm divergence is a novel insight, with broad implications for understanding deep learning generalization and capacity limits.
Limitations
- The analysis assumes specific activation functions (e.g., ReLU) and initialization schemes; applicability to other architectures or training methods remains to be tested.
- Focus is primarily on binary classification tasks; extension to multi-class or regression problems needs further exploration.
- The theoretical results rely on the infinite width limit; finite-width corrections and practical deviations may affect real-world performance.
Future Work
Future directions include extending the framework to multi-class classification, exploring the impact of different regularization techniques, and analyzing finite-width effects. Investigating the behavior under various training algorithms and data distributions will deepen understanding. Additionally, developing adaptive ensemble strategies based on the phase transition insights could optimize model deployment in resource-constrained environments.
AI Executive Summary
Deep neural networks (DNNs) have revolutionized machine learning, yet their generalization mechanisms remain elusive, especially in the overparameterized regime where models contain far more parameters than training points. Empirically, larger networks tend to perform better, contradicting classical statistical theories that predict overfitting. This paradox has motivated recent theoretical advances, notably the neural tangent kernel (NTK) framework, which links infinitely wide networks to kernel methods. Building on this, the present study investigates how finite-size fluctuations influence generalization as the number of parameters, N, varies.
The core discovery is that the output function's fluctuations at initialization scale as N^{-1/4}, leading to a generalization error that decays as N^{-1/2}. This power-law decay continues until N approaches a critical threshold N^{*}, where the network's norm diverges, causing a cusp in test error. This phase transition explains the observed sharp increase in error and provides a theoretical foundation for the cusp phenomenon. Extensive experiments on MNIST and CIFAR datasets validate the theoretical predictions, showing that ensemble averaging of multiple intermediate-sized networks (just beyond N^{*}) can significantly improve performance.
The implications of this work are profound: it offers a unified explanation for the scaling laws of deep learning, guides optimal network sizing, and suggests ensemble strategies for better generalization. By revealing the fundamental role of finite-size fluctuations and the critical point N^{*}, the study bridges empirical observations with rigorous theory, advancing our understanding of deep learning's success and limitations. Future research will explore multi-class extensions, regularization effects, and finite-width corrections, aiming to refine the theoretical framework and enhance practical applications.
Deep Dive
Abstract
Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{*}$. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$ of the neural net output function $f_{N}$ around its expectation $\bar{f}_{N}$. These affect the generalization error $ε_{N}$ for classification: under natural assumptions, it decays to a plateau value $ε_{\infty}$ in a power-law fashion $\sim N^{-1/2}$. This description breaks down at a so-called jamming transition $N=N^{*}$. At this threshold, we argue that $\|f_{N}\|$ diverges. This result leads to a plausible explanation for the cusp in test error known to occur at $N^{*}$. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond $N^{*}$, and averaging their outputs.