Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods

The evolution of deep learning has led to remarkable successes across domains such as computer vision and natural language processing. However, understanding why deep networks generalize well remains a fundamental challenge. Early theoretical efforts focused on over-parameterization and landscape analysis, with approaches like Neural Tangent Kernel (NTK) and mean-field theory providing insights into training dynamics. Despite these advances, limitations persist in explaining high-dimensional nonparametric regression, where kernel methods face the curse of dimensionality. The teacher-student model offers a simplified yet powerful framework to analyze neural network learning, enabling landscape and convergence analysis that bridges optimization and statistical theory.

The core issue is to establish whether shallow neural networks can efficiently learn complex functions in high-dimensional settings, surpassing kernel methods. Existing results often rely on over-parameterization or special initializations, which are impractical. Non-convex loss landscapes pose significant obstacles, with local minima potentially trapping training algorithms. The challenge is to prove that, under realistic conditions, neural networks can achieve minimax optimal rates without exponential width or restrictive assumptions. Addressing this requires a deep understanding of landscape geometry, convergence properties, and the interplay between optimization and generalization.

The paper introduces a two-phase gradient descent approach that leverages Langevin dynamics for global exploration and local strong convexity for precise convergence. This hybrid method allows the neural network to escape poor local minima and approximate the teacher network efficiently. The analysis derives explicit excess risk bounds, showing convergence at the parametric rate O(n^{-1}), which is faster than kernel-based estimators. Key innovations include the landscape-based exploration strategy, the explicit characterization of the error bounds, and the demonstration that finite-width networks can achieve near-global optimality in polynomial time, thus overcoming previous limitations related to over-parameterization.

• �� Model: Teacher-student framework with a finite-width two-layer ReLU network, teacher parameters satisfy spectral bounds.
• �� Training: Two-phase approach—Phase I employs Langevin gradient dynamics with noise and regularization to explore the parameter space; Phase II applies standard gradient descent exploiting local strong convexity.
• �� Analysis: Derive bounds on the empirical and expected risk, utilize landscape geometry, spectral conditions, and concentration inequalities.
• �� Convergence: Show that after sufficient iterations, the student network parameters are close to the teacher, with excess risk decreasing at O(n^{-1}).
• �� Validation: Synthetic data experiments confirm theoretical predictions, comparing error rates with kernel methods across varying sample sizes and dimensions.

Synthetic datasets generated on high-dimensional spheres with Gaussian noise, varying sample size n and input dimension d. Both kernel methods (NTK, random features) and two-phase gradient descent-trained neural networks are evaluated. Hyperparameters such as learning rate, regularization, and noise level are tuned for optimal performance. The experiments measure excess risk convergence, error decay, and robustness to dimensionality. Results show neural networks achieve near-parametric rates, with error decreasing proportionally to 1/n, while kernel methods plateau at slower rates, especially in high dimensions. Ablation studies highlight the importance of the two-phase approach and landscape exploitation.

Empirical results demonstrate that the neural network trained via the proposed method attains an excess risk of O(n^{-1}), significantly outperforming kernel methods, which converge at approximately O(n^{-1/2}). In high-dimensional regimes, the neural network maintains this rate, confirming theoretical predictions. The experiments also reveal that the exploration phase effectively escapes poor local minima, enabling the network to learn the teacher function efficiently. These findings validate the landscape-based analysis and establish the superiority of deep models in nonparametric regression tasks.

The methodology applies to high-dimensional function approximation, enabling efficient learning in fields like genomics, image analysis, and financial modeling. It requires only a shallow network with appropriate regularization and noise injection, making it computationally feasible. The approach can be integrated into existing training pipelines to improve generalization and convergence speed, especially in settings where kernel methods are limited by dimensionality. Its robustness and theoretical guarantees make it suitable for deployment in real-world, large-scale problems.

The analysis assumes specific spectral conditions on teacher parameters, which may not hold universally. The method relies on idealized noise and regularization parameters, requiring careful tuning in practice. Extending the results to deeper networks involves additional complexity, as landscape geometry becomes more intricate. Moreover, the current framework does not account for practical issues like data heterogeneity, non-Gaussian noise, or computational constraints in large-scale training. Future work should address these gaps to enhance applicability.