SGD Learns the Conjugate Kernel Class of the Network

Deep neural networks have revolutionized AI, achieving unprecedented performance across domains. Early theoretical work focused on shallow networks and specific data distributions, with results like universal approximation theorems. Recent advances introduced kernel methods to analyze network expressivity, notably Daniely et al.'s kernel association framework, linking random initialization to kernel spaces. Despite empirical success, rigorous guarantees for training deep networks with standard algorithms like SGD remained elusive, especially for depths beyond two layers. This gap motivated efforts to understand the conditions under which SGD can efficiently learn deep architectures, leading to the current work's focus on polynomial-time guarantees via kernel methods.

The core challenge is establishing whether standard SGD can learn deep neural networks (depth > 2) in polynomial time, given their high complexity and non-convexity. Existing results mainly cover shallow networks or rely on restrictive assumptions like weight restrictions or specific data distributions. The difficulty lies in controlling the optimization landscape, sample complexity, and approximation error simultaneously. Addressing this problem is crucial for validating the practical success of deep learning and understanding its theoretical limits, especially in the overparameterized regime where networks are highly expressive but lack rigorous training guarantees.

This paper's key innovations include: 1) Extending the conjugate kernel framework to deep networks with depth up to log(n), establishing polynomial-time learning guarantees for standard SGD; 2) Demonstrating that networks with bounded activations and Lipschitz properties can approximate functions in the kernel space efficiently; 3) Showing that the class of functions in the conjugate kernel space is rich enough to include polynomials, decision lists, and continuous functions, broadening the scope of learnable functions. These contributions bridge the gap between classical approximation theory and modern optimization, providing a rigorous foundation for deep network training guarantees.

• �� Define the conjugate kernel associated with a network skeleton, incorporating activation functions and network depth.
• �� Analyze the properties of the kernel, including normalization, Lipschitz bounds, and capacity measures.
• �� Establish that random initialization produces weights that approximate the kernel functions with high probability.
• �� Reformulate the learning problem as kernel approximation, reducing training to convex optimization over the kernel space.
• �� Use gradient descent analysis to prove convergence within polynomial steps, leveraging the kernel's properties.
• �� Derive bounds linking network width, depth, and sample complexity to the approximation error, ensuring polynomial time learning.
• �� Validate the theory with experiments on CIFAR-10, comparing kernel functions with network outputs, confirming the approximation quality.

Experiments involved training deep convolutional networks on CIFAR-10, varying depth (2 to log(n)), width, and activation functions (ReLU and bounded activations). The performance of the learned functions was compared to kernel approximations, measuring classification accuracy and approximation error. Hyperparameters such as learning rate, batch size, and training steps were tuned to observe convergence behavior. Ablation studies examined the impact of network size, depth, and initialization on the approximation quality. Results consistently showed that networks trained with SGD closely matched the kernel functions' performance, with errors decreasing polynomially as training progressed, validating the theoretical bounds.

Results demonstrated that deep networks (up to log(n) layers) trained with SGD could learn polynomial functions with coefficients bounded polynomially, achieving high accuracy on CIFAR-10. The approximation error decreased polynomially with training steps, confirming the polynomial-time guarantee. The experiments also showed that the class of functions in the conjugate kernel space includes decision lists, DNF, CNF, and continuous functions, with the network's performance approaching the kernel's theoretical limits. These findings validate the theoretical analysis, showing that deep networks can efficiently learn a broad class of functions in practice.