Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks

The evolution of neural network theory has transitioned from shallow models to deep architectures, with infinite-width limits like NTK and Gaussian process models providing foundational insights. Early works such as Jacot et al. (2018) introduced the neural tangent kernel, enabling linearized training analysis. Subsequent studies, including Chizat & Bach (2018), explored mean field regimes where feature learning occurs. However, these models often assume infinite width, neglecting finite-size effects that introduce bias and variance in predictions. Recent efforts have begun to quantify finite-width corrections, but a comprehensive understanding of how these fluctuations influence training dynamics, especially in deep networks, remains elusive. This paper builds on these foundations, aiming to systematically quantify the stochastic fluctuations of kernels and predictions arising from finite network widths, and how feature learning modulates these effects.

While infinite-width models simplify analysis, real neural networks are finite, leading to fluctuations in kernels and predictions that impact training stability and generalization. Existing theories lack a unified framework to quantify these finite-size effects dynamically during training, especially in deep architectures. The core challenge is to develop a theoretical model that captures the coupled evolution of kernel fluctuations and feature learning, providing accurate predictions of bias and variance corrections. Addressing this gap is crucial for understanding the practical limitations of neural networks, optimizing hyperparameters, and designing architectures that leverage finite-width effects for improved performance.

This work introduces a novel application of DMFT to derive explicit O(1/√N) fluctuation formulas for kernels and predictions during training, extending classical mean field analysis. It incorporates feature learning dynamics into the fluctuation analysis, revealing how feature evolution dynamically reduces variance and enhances SNR. The framework captures layer-wise kernel variance accumulation and provides self-consistent equations validated through numerical simulations. This approach bridges the gap between infinite-width idealizations and finite networks, offering new theoretical tools to analyze training dynamics, generalization, and robustness in realistic neural architectures.

• �� Derive the DMFT equations for infinite-width networks, defining key order parameters such as kernels and predictions. • Expand these equations to include finite-width fluctuations, leading to O(1/√N) correction terms. • Formulate self-consistent equations for the covariance (propagator) of these fluctuations, incorporating feature learning effects. • Solve the equations numerically for simple regimes (lazy training) and more complex regimes (rich feature learning). • Validate predictions with simulations of two-layer ReLU networks and deep linear models trained on CIFAR-10. • Analyze the impact of network depth, training set size, and learning rate on fluctuation dynamics.

Experiments involve training two-layer ReLU networks with width N=64 to N=256 on CIFAR-10, measuring prediction variance, kernel fluctuations, and bias corrections over training epochs. Numerical solutions of the self-consistent equations are compared against ensemble simulations, showing high accuracy in predicting finite-width effects. Additional tests include deep linear networks and CNNs, examining how kernel variance accumulates across layers and how feature learning mitigates this. Hyperparameters such as learning rate and initialization variance are varied to assess their influence on fluctuations. Results consistently demonstrate the theory's predictive power and practical relevance.

Theoretical predictions of O(1/N) bias and variance corrections match empirical measurements across architectures. Feature learning reduces kernel variance and improves SNR, leading to better generalization. In deep networks, layer-wise kernel variance increases with depth, but feature learning dynamically suppresses this accumulation. Large learning rates induce boundary effects (edge of stability), which are accurately captured by the theory. Empirical data from CNNs trained on CIFAR-10 show finite-width corrections significantly impact training dynamics, validating the importance of accounting for fluctuations in practical models.

The framework guides optimal network initialization, training rate selection, and architecture design by quantifying finite-width effects. It informs strategies to mitigate variance-induced instability, improve robustness, and enhance generalization, especially in resource-constrained settings. The insights are applicable to a broad range of models, including CNNs, RNNs, and transformers, and can influence hyperparameter tuning, model compression, and transfer learning. Long-term, the theory can underpin the development of adaptive training algorithms that dynamically control fluctuations for improved performance.

The analysis relies on Gaussian assumptions and high-dimensional asymptotics, which may not fully capture finite-sample complexities or non-Gaussian initializations. Empirical validation is limited to specific architectures and datasets, requiring further testing on larger, more diverse models. Computational complexity of solving self-consistent equations increases with network size and depth, posing challenges for real-time applications. Extending the framework to include optimization algorithms and regularization effects remains an open problem, as does understanding finite-width effects in non-i.i.d. data scenarios.