Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy

Kernel methods have long been a cornerstone of non-parametric statistics, with foundational work by Caponnetto and De Vito (2007) establishing spectral capacity conditions. Recent advances, such as Belkin et al. (2018), linked neural networks to kernel regimes via the neural tangent kernel, sparking renewed interest. Kaplan et al. (2020) observed power-law scaling laws in large models, but these studies largely assume isotropic or simple data distributions. In contrast, real-world data often exhibits complex geometric structures, such as power-law spectra in images or text, which significantly influence kernel eigenvalues. Wortsman and Loureiro (2025) initiated spectral analysis for anisotropic Gaussian inputs, revealing the importance of input geometry. Despite progress, a comprehensive understanding of how anisotropic spectral decay impacts learning curves remains elusive, motivating this work.

Existing theories predominantly focus on isotropic or idealized data, leaving a gap in understanding the effects of realistic anisotropic structures. Specifically, how power-law decay in input covariance influences the kernel spectrum, bias, variance, and ultimately the generalization error in high dimensions is not well-understood. The challenge lies in deriving explicit asymptotic formulas that incorporate the input geometry, spectral decay, and target structure simultaneously. Addressing this problem is critical for developing models that are robust to data heterogeneity, and for understanding phase transitions in learning behavior as the anisotropy varies. The difficulty is compounded by the need for precise spectral bounds and the complexity of high-dimensional asymptotics.

This work introduces a novel framework combining spectral analysis, random matrix theory, and asymptotic risk formulas tailored for power-law anisotropic Gaussian inputs. Key innovations include: 1) deriving the asymptotic spectrum of polynomial kernels under power-law decay, 2) formulating the kernel state equation capturing the interplay between spectral decay and sample complexity, 3) revealing phase transitions in variance and bias depending on anisotropy strength, and 4) providing explicit risk formulas that interpolate between high- and low-dimensional regimes. These contributions extend prior spectral bounds to precise asymptotic equivalences, bridging the gap between spectral theory and statistical learning in complex data environments.

• �� Assume high-dimensional limit d→∞, with sample size n scaling as n=Θ(d^κ).
• �� Model input data as Gaussian with covariance Σ exhibiting power-law eigenvalue decay σ_j = r_α(d)^−1 j^−α.
• �� Derive the spectral distribution of the polynomial kernel using random matrix theory, establishing asymptotic eigenvalue formulas.
• �� Formulate the kernel state equation linking spectral decay, effective regularization, and risk components.
• �� Solve the state equation for different α regimes, identifying phase transitions in variance and bias.
• �� Use bias-variance decomposition to derive explicit asymptotic expressions for excess risk, incorporating target alignment effects.

• �� Generate synthetic anisotropic Gaussian data with varying α to validate spectral formulas.
• �� Implement polynomial kernel ridge regression, tuning regularization λ, across different sample complexities κ.
• �� Measure empirical learning curves, bias, and variance, comparing with theoretical predictions.
• �� Analyze the impact of target alignment with principal directions, especially for single-index models.
• �� Conduct ablation studies varying α and target decay rates to explore phase transition behaviors.

• �� The derived spectral formulas accurately predict the eigenvalue decay, confirming the asymptotic equivalence for various α.
• �� Variance peaks at integer κ diminish as α increases, with the peaks becoming negligible in the strong anisotropy regime.
• �� Bias exhibits a sharp transition at fractional sample complexities, with power-law decay above a threshold, matching classical rates.
• �� Effective dimension stabilizes for α>1, leading to variance saturation and exponential decay, simplifying the learning dynamics.
• �� Target alignment influences the bias transition point, demonstrating the geometric impact on learning efficiency.

• �� The results guide the design of kernel functions tailored to data geometry, improving generalization in high-dimensional tasks like image recognition and natural language processing.
• �� They enable practitioners to predict the sample complexity needed for desired accuracy based on input spectral properties.
• �� Long-term, insights from this work can inform neural network scaling laws and architecture choices, especially in heterogeneous data environments.

• �� The analysis assumes Gaussian inputs with power-law spectra, which may not fully capture real-world data complexities.
• �� Focus is on polynomial kernels; extension to other kernels like Gaussian RBF remains future work.
• �� Asymptotic results may not directly translate to finite-sample scenarios, requiring empirical validation and correction.