Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime
Applying Takeuchi Information Criterion (TIC) to evaluate DNN generalization near NTK regime; experiments show high correlation with actual gaps.
Key Findings
Methodology
This study leverages TIC, combining Hessian and gradient covariance matrices, to theoretically and empirically analyze its effectiveness in the NTK regime. By training over 5000 models across 12 architectures, the authors estimate TIC values using approximations like Fisher information, diagonal, and block-diagonal methods. Theoretical derivations confirm TIC’s validity under NTK assumptions, with experiments validating strong correlation (coefficients >0.8) between TIC and generalization gaps in this regime. The approach balances computational efficiency and accuracy, enabling large-scale application.
Key Results
- Results demonstrate that in the NTK regime, TIC estimates correlate strongly with actual generalization gaps (correlation >0.8). For example, in VGG-16 models, TIC's correlation coefficient with test error difference reached 0.85, outperforming validation loss. Approximate methods like Fisher-based estimation maintained high accuracy with reduced computational cost. Outside NTK conditions, correlation diminishes sharply, indicating limited applicability. Additionally, TIC-based trial pruning in hyperparameter tuning outperformed traditional validation loss, effectively avoiding premature pruning of promising models.
- Experiments also reveal that wider networks and residual connections improve TIC’s predictive power, aligning with theoretical expectations. The approximation strategies (diagonal, block-diagonal) offer practical trade-offs, enabling scalable computation for models with millions of parameters. The study confirms that TIC can serve as a reliable indicator for model selection, especially when models are sufficiently overparameterized and close to NTK assumptions.
- Furthermore, the analysis highlights the importance of network architecture and training conditions. When the ratio of parameters to data points is large, and the NTK remains stable during training, TIC provides meaningful estimates. Conversely, in feature-learning regimes where NTK assumptions break down, the correlation drops, suggesting the need for alternative measures like WAIC or Bayesian approaches for singular models.
Significance
This work bridges classical statistical information criteria with modern deep learning theory, providing a rigorous foundation for using TIC as a generalization measure. It offers a scalable, theoretically justified tool for model evaluation and hyperparameter tuning, addressing the long-standing challenge of understanding deep neural network generalization. The empirical validation across diverse architectures and datasets demonstrates its practical relevance, especially in large-scale settings. By clarifying the conditions under which TIC is effective, the study advances both theoretical insights and applied methodologies, fostering more reliable and interpretable deep learning models.
Technical Contribution
The main technical contribution lies in adapting TIC, traditionally used in regular statistical models, to the singular, high-dimensional setting of DNNs within the NTK regime. The authors derive conditions under which the Hessian, Fisher, and GGN matrices become equivalent, enabling efficient approximation strategies. They develop scalable algorithms—Monte Carlo estimation, diagonal and block-diagonal approximations, Hutchinson’s method—to compute TIC for models with millions of parameters. Theoretical proofs confirm the validity of these approximations under NTK assumptions, and extensive experiments validate their effectiveness. This work extends classical information criteria into the deep learning domain, providing a new theoretical and practical framework for generalization assessment.
Novelty
This is the first comprehensive study to theoretically justify and empirically validate TIC as a measure of generalization in deep neural networks, specifically within the NTK regime. Unlike prior work limited to small networks, this research introduces scalable approximation algorithms suitable for large models like VGG-16 and ResNet-50. It also clarifies the regime conditions (overparameterization, NTK stability) necessary for TIC’s effectiveness, bridging classical statistical theory with modern deep learning. The integration of multiple approximation techniques and extensive empirical validation marks a significant advancement in the field.
Limitations
- TIC’s reliability heavily depends on the model being in the NTK regime; outside this regime, the correlation with generalization gaps diminishes sharply, limiting its universal applicability.
- Despite approximation strategies, computing TIC for very large models remains computationally intensive, especially when the NTK changes significantly during training.
- Current methods assume static training conditions; dynamic training processes, such as feature learning outside NTK assumptions, are not well captured, requiring further research.
Future Work
Future research will explore extending TIC’s theoretical foundation beyond NTK assumptions, incorporating training dynamics and feature learning effects. Developing adaptive approximation algorithms that account for NTK stability changes during training is a key direction. Additionally, integrating Bayesian methods like WAIC for singular models and exploring real-time TIC estimation during training could further enhance its practical utility. Cross-domain applications, such as transfer learning and reinforcement learning, also present promising avenues for extending this framework.
AI Executive Summary
Deep neural networks (DNNs) have revolutionized many fields, yet understanding their generalization remains a fundamental challenge. Traditional metrics like validation loss often fall short in capturing the true predictive power of complex models, especially as architectures grow larger and more overparameterized. To address this, recent theoretical advances have pointed towards information-theoretic measures, notably the Takeuchi Information Criterion (TIC), as promising tools for assessing generalization. This study rigorously investigates TIC’s applicability to DNNs, particularly within the neural tangent kernel (NTK) regime where the network behaves approximately linearly around initialization.
The authors derive the theoretical conditions under which TIC provides an unbiased estimate of the generalization gap, leveraging the properties of the NTK to justify the regularity assumptions needed. They develop practical approximation algorithms—Monte Carlo, diagonal, and block-diagonal methods—to compute TIC efficiently for large-scale models like VGG-16 and ResNet-50. Extensive experiments across diverse datasets (MNIST, CIFAR-10, TinyMNIST) and architectures demonstrate that TIC correlates strongly with actual generalization gaps when the network operates close to the NTK regime, with correlation coefficients exceeding 0.8. Notably, TIC-based hyperparameter trial pruning outperforms traditional validation loss, effectively identifying promising models.
However, outside the NTK regime, the correlation diminishes, highlighting the method’s limitations. The findings suggest that TIC can serve as a reliable, scalable tool for model evaluation and selection in overparameterized settings, providing new insights into the theoretical underpinnings of deep learning generalization. This work bridges classical statistical theory with modern deep learning practice, opening avenues for more interpretable and efficient model optimization in the future.
Deep Dive
Abstract
Generalization measures have been studied extensively in the machine learning community to better characterize generalization gaps. However, establishing a reliable generalization measure for statistically singular models such as deep neural networks (DNNs) is difficult due to their complex nature. This study focuses on Takeuchi's information criterion (TIC) to investigate the conditions under which this classical measure can effectively explain the generalization gaps of DNNs. Importantly, the developed theory indicates the applicability of TIC near the neural tangent kernel (NTK) regime. In a series of experiments, we trained more than 5,000 DNN models with 12 architectures, including large models (e.g., VGG-16), on four datasets, and estimated the corresponding TIC values to examine the relationship between the generalization gap and the TIC estimates. We applied several TIC approximation methods with feasible computational costs and assessed the accuracy trade-off. Our experimental results indicate that the estimated TIC values correlate well with the generalization gap under conditions close to the NTK regime. However, we show both theoretically and empirically that outside the NTK regime such correlation disappears. Finally, we demonstrate that TIC provides better trial pruning ability than existing methods for hyperparameter optimization.