Annihilation of Spurious Minima in Two-Layer ReLU Networks

The evolution of deep learning has been marked by a growing complexity of neural architectures and non-convex optimization landscapes. Early theoretical efforts focused on the properties of the neural tangent kernel (NTK), mean-field limits, and infinite-width analyses, which provided insights into convergence and generalization. However, these approaches often neglect finite-size effects and the role of symmetry in the loss landscape. Recent research highlights the importance of symmetry structures, such as permutation invariance, in shaping critical points. Techniques from algebraic geometry and representation theory have been employed to analyze these structures, but a comprehensive understanding of how over-parameterization influences the elimination of spurious minima remains lacking. This paper addresses this gap by providing a rigorous spectral and geometric analysis of finite two-layer ReLU networks, emphasizing the role of symmetry breaking.

The key challenge is understanding why large neural networks trained with gradient descent often escape poor local minima and converge to global minima despite the non-convexity. Traditional analyses focus on the infinite-width limit, which simplifies the landscape but does not fully explain finite network behavior. The core problem is to mathematically characterize how increasing the number of neurons affects the structure and stability of critical points, especially the symmetric spurious minima. This involves analyzing the loss landscape's geometry, the spectral properties of the Hessian, and the mechanisms by which symmetry breaking facilitates optimization. Addressing this problem is crucial for designing scalable, efficient training algorithms and understanding the empirical success of over-parameterization.

The paper introduces a novel framework combining symmetry analysis, algebraic geometry, and spectral theory to study finite neural networks. Key innovations include: 1) employing Puiseux series to describe the asymptotic behavior of minima; 2) proving the existence of descent directions via Cauchy interlacing, which shows how adding neurons destroys symmetric minima; 3) deriving explicit spectral estimates for the Hessian at critical points, revealing how eigenvalues shift with network width. These methods extend beyond previous infinite-width or mean-field approaches, providing a detailed geometric and spectral picture of the loss landscape in realistic finite models. The approach offers new insights into the role of symmetry breaking in deep learning optimization.

• �� Define families of critical points with specific symmetry properties using group actions. • Use Puiseux series expansions to analyze the asymptotic behavior of minima as network width increases. • Apply Cauchy interlacing theorem to identify descent directions in subspaces associated with symmetry structures. • Derive spectral estimates of the Hessian by expressing eigenvalues in terms of gradient entries, enabling precise analysis at critical points. • Employ algebraic geometry tools to analyze the stability and multiplicity of critical points. • Conduct numerical simulations to validate spectral predictions and analyze the effect of target perturbations. • Investigate how increasing neurons alters the eigenvalue distribution, transforming minima into saddles, supported by spectral and geometric analysis.

The experiments utilize synthetic datasets generated by symmetric target networks, with inputs drawn from Gaussian distributions. The number of neurons varies around the input dimension (d, d+1, d+2). Loss values and Hessian spectra are computed at critical points identified via algebraic methods. Puiseux series expansions are used to analyze the asymptotic behavior of minima as network width increases. Numerical spectral estimates validate the theoretical predictions, showing eigenvalues' concentration and shifts consistent with the symmetry-breaking mechanism. Additional experiments perturb the target functions to test spectral stability, confirming the robustness of the theoretical framework. The results demonstrate that adding neurons systematically eliminates symmetric spurious minima, aligning with the spectral and geometric analysis.

Adding a small number of neurons (e.g., two) effectively transforms symmetric spurious minima into saddle points, facilitating optimization. Spectral analysis reveals that eigenvalues associated with these minima concentrate in low-dimensional subspaces, with some eigenvalues growing linearly with input dimension d. Puiseux series expansions accurately describe the asymptotic behavior of minima, confirming that over-parameterization smooths the loss landscape. The spectral stability under target perturbations indicates robustness, and the existence of descent directions in specific subspaces validates the theoretical mechanism for escaping poor local minima. These results collectively explain the empirical success of large neural networks in training.

The insights from this study can guide the design of training algorithms that leverage symmetry breaking to escape local minima more efficiently. It provides a theoretical basis for adaptive network expansion strategies, dynamically adding neurons based on spectral properties. The framework can be extended to multi-layer architectures, improving convergence guarantees and robustness. In industry, these results support the development of scalable training pipelines for large models, especially in settings where symmetry-induced traps hinder optimization. Long-term, this work paves the way for automated neural architecture search methods that incorporate spectral and geometric considerations for optimal network design.

The analysis is primarily focused on two-layer ReLU networks with symmetric targets, limiting direct applicability to more complex, multi-layer models. The spectral estimates rely on idealized assumptions, such as perfect symmetry and noiseless data, which may not hold in real-world scenarios. Numerical stability issues may arise in high-dimensional spectral computations, requiring further algorithmic development. Additionally, the impact of data distribution heterogeneity and non-symmetric targets remains to be explored, posing challenges for practical deployment. Future work should address these limitations by extending the theoretical framework and validating it on real datasets.