On the Power and Limitations of Random Features for Understanding Neural Networks

The rapid advancement of deep learning has transformed AI, with architectures like CNNs and Transformers achieving unprecedented performance. Early theories explained success via over-parameterization, which eases optimization and enables memorization. Random features emerged as a promising framework, approximating neural networks by fixed, randomly initialized mappings. Prior works like Rahimi and Recht's Random Kitchen Sinks demonstrated their power in kernel approximation, while recent studies linked over-parameterization to neural tangent kernels. Despite these insights, the fundamental expressive limits of random features, especially in high dimensions, remained unclear. Understanding these limits is crucial for explaining why deep networks generalize well despite their size and complexity.

The core challenge is to theoretically explain the expressive power of neural networks, particularly how they learn complex functions efficiently. While empirical evidence shows deep networks outperform shallow or fixed feature models, existing theories like the neural tangent kernel or random features fall short in explaining the full capacity. Specifically, the question is whether random features can account for the ability to learn highly non-linear functions, such as single neurons with ReLU activation, in high-dimensional spaces. The problem is compounded by the fact that random features are fixed mappings, and their capacity to approximate complex functions may require exponential parameter growth, which is computationally infeasible. Clarifying these limitations is essential for advancing the theoretical understanding of deep learning.

The paper's key innovations include: 1) a rigorous proof that approximating a single ReLU neuron with Gaussian inputs requires exponential numbers of features or weights, establishing a fundamental expressiveness bottleneck; 2) the application of polynomial approximation techniques, such as Legendre polynomial expansions, to quantify the capacity of random features; 3) a novel combination of concentration inequalities and spectral analysis to derive explicit bounds on the number of features needed for function approximation; 4) a constructive demonstration that one-hidden-layer networks can learn low-degree polynomials efficiently, contrasting with the limitations of random features. These contributions provide a new, mathematically rigorous perspective on the limitations of fixed random mappings in high-dimensional neural network models.

• �� Model neural networks as random feature mappings, employing Taylor series to express activation functions as infinite polynomials.
• �� Use Legendre polynomial expansions to construct functions g(w) that approximate target polynomials within the random feature integral framework.
• �� Apply concentration of measure inequalities to bound deviations of finite random feature sums from their expectations.
• �� Derive explicit bounds on the number of features r and weight magnitudes, showing exponential growth is necessary for accurate approximation.
• �� Prove that one-hidden-layer networks can learn low-degree polynomials with polynomially many parameters, using over-parameterized SGD and initialization schemes.
• �� Integrate these results into a comprehensive proof demonstrating the fundamental limits of random features in high-dimensional function approximation.

The work primarily relies on mathematical proofs, supplemented by numerical simulations validating the exponential parameter requirement for approximating single neurons. Simulations involve varying input dimension d, number of features r, and weight magnitudes, confirming the theoretical bounds. Additional experiments demonstrate the ability of over-parameterized networks to learn low-degree polynomials, aligning with the positive results. These simulations use Gaussian input distributions, ReLU and polynomial activation functions, and standard SGD optimization. The experimental results consistently support the theoretical claims, illustrating the exponential scaling barrier and the polynomial learnability of certain functions by neural networks.

The main quantitative findings include: (1) random features need at least exp(Ω(d)) parameters to approximate a single ReLU neuron with Gaussian inputs; (2) one-hidden-layer networks can learn polynomials of bounded degree efficiently, with polynomially many parameters; (3) random features are fundamentally limited in high dimensions, unable to explain the learnability of simple neurons without exponential resources. These results establish clear bounds on the expressive power of fixed random mappings, highlighting the necessity of learned, hierarchical representations in deep networks.

The insights inform the design of neural architectures, emphasizing the importance of learned features over fixed random mappings. They guide practitioners in understanding the limitations of random feature-based methods, especially for high-dimensional data. The results also impact the development of theoretical models for deep learning, suggesting that effective generalization relies on adaptive, data-dependent representations. In industry, these findings support efforts to optimize training strategies and model complexity, ensuring scalable and efficient learning in real-world applications such as image recognition, natural language processing, and reinforcement learning.

The analysis assumes Gaussian input distributions, which may not reflect real data complexities. It focuses on shallow networks, leaving the behavior of deep architectures less understood. The exponential parameter requirement limits practical deployment, indicating the need for alternative approaches that can achieve expressive power with fewer resources. Further research is needed to extend these bounds to broader data distributions and deeper models, and to develop methods that circumvent these theoretical constraints.