Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization
This study analyzes optimizer-dependent training dynamics via Hessian eigenvector displacement and localization, revealing SGD stabilizes directions while Adam induces eigenvector reorganization.
Key Findings
Methodology
Using multilayer perceptrons trained on classification tasks, the research tracks Hessian eigenvector evolution through displacement and inverse participation ratio (IPR). Employing Lanczos algorithms for efficient eigenvector computation, it compares the effects of SGD, Adam, and SAM optimizers. The study benchmarks against a random Hessian null model, revealing how each optimizer influences the geometry of the loss landscape during mini-batch training, with detailed analysis of eigenvector stability, displacement, and localization over training steps.
Key Results
- SGD shows a progressive stabilization of the leading eigenvector, with displacement decreasing and localization diminishing as training proceeds, indicating convergence toward flatter regions. Conversely, Adam exhibits persistent eigenvector reorganization, with displacement remaining high and IPR increasing to about 0.1, signifying strong localization on a small subset of parameters. These behaviors are consistent across datasets (MNIST, FashionMNIST), hyperparameters, and batch sizes, demonstrating fundamental differences in how optimizers explore the loss landscape.
- Analysis of eigenvector alignment with gradients and parameter updates reveals SGD maintains high alignment (>0.8), stabilizing the dominant directions, while Adam's alignment remains below 0.1, reflecting ongoing eigenvector rotation. This suggests Adam promotes exploration of different curvature directions, potentially aiding in escaping sharp minima or saddle points.
- Eigenvector localization analysis shows Adam induces a localized eigenvector structure, with few parameters dominating the curvature directions, whereas SGD promotes delocalization. These phenomena are robust and indicate that optimizer choice significantly impacts the geometric exploration and parameter sensitivity during training.
Significance
This work provides a novel geometric perspective on training dynamics, highlighting how optimizer choice influences the evolution of curvature directions in high-dimensional parameter spaces. Understanding eigenvector behavior offers insights into generalization, robustness, and convergence properties of neural networks. The findings suggest that eigenvector reorganization and localization are key mechanisms underlying optimizer performance, guiding future development of algorithms that leverage these geometric features to improve training efficiency and model robustness.
Technical Contribution
The study introduces quantitative metrics—displacement and IPR—for tracking Hessian eigenvector dynamics during training. It combines efficient Lanczos-based eigenvector computation with a random Hessian baseline, enabling detailed analysis of how different optimizers influence the curvature directions. This approach advances the theoretical understanding of training geometry, revealing optimizer-dependent eigenvector reorganization and localization phenomena, and provides a framework for future investigations into high-dimensional loss landscapes.
Novelty
This is the first comprehensive analysis of individual Hessian eigenvector evolution during mini-batch training, contrasting optimizer effects at a fine-grained level. Unlike prior work focusing on eigenvalues or subspace stability, this study explicitly tracks eigenvector displacement and localization, uncovering distinct behaviors—stability in SGD versus persistent reorganization and localization in Adam—thus offering new insights into the geometric mechanisms of optimization in deep learning.
Limitations
- The analysis is limited to small-scale multilayer perceptrons; complex architectures like ResNets or transformers may exhibit different eigenvector dynamics, requiring further validation.
- Computational costs of eigenvector tracking increase with model size; developing scalable approximations is necessary for large-scale networks.
- The study focuses on specific hyperparameters; broader hyperparameter sweeps and task diversity are needed to generalize findings across different training regimes.
Future Work
Future research will explore the relationship between eigenvector dynamics and generalization, robustness, and convergence stability. Extending analysis to larger, more complex architectures and different tasks will validate the universality of these phenomena. Additionally, integrating eigenvector localization insights into optimizer design and model pruning strategies could lead to more efficient training algorithms and compressed models. Investigating the connection between eigenvector reorganization and edge-of-stability phenomena may also deepen theoretical understanding.
AI Executive Summary
Deep neural network training involves navigating a high-dimensional, non-convex loss landscape, where understanding the local geometry is crucial for improving optimization and generalization. Traditional spectral analysis of the Hessian matrix provides insights into sharpness and flatness, but often overlooks the directional dynamics of eigenvectors. This study advances the field by explicitly tracking the evolution of leading Hessian eigenvectors during mini-batch training on classification tasks with multilayer perceptrons.
Using metrics such as eigenvector displacement and inverse participation ratio (IPR), the authors compare the effects of different optimizers—SGD, Adam, and SAM—on the geometry of the loss landscape. The results reveal that SGD tends to stabilize the dominant curvature directions over time, with eigenvector displacement decreasing and localization diminishing, indicating convergence toward flatter, more stable regions. In contrast, Adam exhibits continuous eigenvector reorganization, with persistent displacement and increased localization, suggesting a different exploration strategy that emphasizes localized parameter sensitivity.
These geometric insights have profound implications. The eigenvector behaviors correlate with optimizer-specific trajectories, affecting how models escape sharp minima or saddle points. The localized eigenvectors in Adam imply that a small subset of parameters dominates the curvature directions, which could influence pruning and robustness strategies. The findings bridge the gap between spectral properties and training dynamics, offering a new framework for understanding and improving deep learning optimization.
Despite these advances, the study's scope is limited to small models, and computational challenges remain for scaling to larger architectures. Future work aims to extend these analyses, explore the relationship between eigenvector dynamics and generalization, and leverage these insights for designing more robust and efficient training algorithms. Overall, this research provides a nuanced, geometric perspective on neural network training, opening avenues for theoretical and practical innovations in deep learning.
Deep Dive
Abstract
Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors identify which parameters generate that curvature. In this work, we study how the leading Hessian eigenvectors evolve during training and how they affect the learning trajectories. We track the training dynamics of multilayer perceptrons on a classification problem and measure eigenvector dynamics through two complementary statistics: (i) displacement over time, inspired by analyses of glassy systems, and (ii) localization via the inverse participation ratio. The metrics are compared against a random null model of the Hessian induced by the architecture. Our results reveal clear optimizer-dependent behaviour. SGD leads to progressively more stable leading curvature directions, while Adam exhibits substantially stronger reorganization of eigenvectors throughout training. We also observe a localization phenomenon under Adam, where a small subset of parameters contributes disproportionately to the leading curvature directions. These results suggest that Hessian eigenvector dynamics capture key differences in optimizer behaviour and the resulting training trajectories.