Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
This paper derives exponential approximation bounds for analytic functions using ReLU networks, highlighting depth's dominance over width in the (N,L) parameterization.
Key Findings
Methodology
The authors construct multi-layer ReLU networks leveraging polynomial approximation, Legendre expansions, and complex analysis. They utilize Bernstein ellipse extensions to control the approximation error of infinitely differentiable functions. By designing networks that approximate powers, multivariate multiplication, and polynomials, they analyze the error decay relative to network parameters. The approach combines classical polynomial approximation with deep network architectures, deriving exponential convergence rates. The core technical innovation is the refined network construction that balances approximation accuracy and network complexity, establishing near-optimal bounds through matching lower bounds.
Key Results
- Under the condition L^κ+α ≤ N ≤ e^{L^β}, the approximation error achieves O(N^{-C L^τ}), with τ depending on κ. When κ=d, the error decay is exponential in depth, surpassing polynomial rates for finite smoothness functions. Numerical experiments confirm that increasing depth yields exponential error reduction, with the rate significantly faster than traditional polynomial approximation. The constructed networks approximate high-dimensional analytic functions with near-optimal efficiency, demonstrating the critical role of depth.
- The upper bounds closely match the derived lower bounds, especially in the κ=d regime, indicating near-minimax optimality. For example, in 2D, the error decreases exponentially with depth, outperforming classical polynomial methods by orders of magnitude. The network constructions for powers, multiplication, and Legendre polynomials underpin these results, providing a rigorous theoretical foundation for deep neural approximation.
- The methodology advances the understanding of how deep architectures exploit the infinite smoothness of analytic functions, establishing a framework that can be extended to other high-smoothness function classes and activation functions.
Significance
This work fundamentally shifts the understanding of neural network approximation capabilities, showing that depth plays a more vital role than width for analytic functions. It provides a rigorous mathematical foundation for the empirical success of deep networks in high-precision scientific computing tasks. The exponential approximation rate signifies that deep networks can efficiently encode complex, infinitely smooth functions, which was previously unquantified. These insights influence the design of neural architectures for high-accuracy tasks, including PDE solvers, quantum physics simulations, and high-dimensional data modeling. The theoretical results bridge the gap between classical approximation theory and modern deep learning, offering a new lens to analyze the power of depth in neural networks.
Technical Contribution
The paper introduces a novel construction of neural networks approximating powers, multivariate multiplication, and Legendre polynomial expansions. It combines polynomial approximation theory with deep network architectures, employing Bernstein ellipse techniques to control errors in the complex domain. The derived exponential convergence bounds are supported by matching lower bounds, establishing near-optimality. The approach generalizes previous polynomial approximation results to the infinite smoothness setting, revealing the dominant role of depth. This framework opens avenues for rigorous analysis of high-precision approximation in deep learning, with potential extensions to other activation functions and function classes.
Novelty
This is the first comprehensive analysis demonstrating exponential approximation rates for analytic functions via deep ReLU networks within the (N,L) parameterization. It emphasizes the critical role of depth, contrasting with prior finite smoothness results that treat depth and width symmetrically. The integration of complex analysis, polynomial approximation, and neural network design constitutes a significant innovation, providing a new theoretical paradigm for understanding deep learning's approximation power for infinitely smooth targets.
Limitations
- The results rely on the assumption that target functions admit a holomorphic extension within Bernstein ellipses, which may not hold for functions with singularities or limited analyticity.
- Constructing networks with the prescribed complexity is theoretically feasible but may be computationally intensive in practice, especially in high dimensions.
- The analysis focuses on approximation bounds, leaving open questions about training dynamics, stability, and generalization in practical settings.
Future Work
Future research could extend these results to broader classes of smooth functions, including Gevrey or Sobolev spaces, and explore the impact of different activation functions. Investigating the training algorithms that realize these approximation bounds efficiently remains an open challenge. Additionally, applying this theoretical framework to real-world scientific problems, such as PDE solutions or quantum simulations, could significantly enhance the practical relevance of deep neural networks in high-precision tasks.
AI Executive Summary
Deep neural networks have revolutionized many fields, yet their theoretical approximation capabilities, especially for highly smooth functions, remain an active research frontier. Traditional theories primarily focus on the total number of parameters or network width, often neglecting the critical influence of depth. This paper addresses this gap by establishing exponential approximation bounds for analytic functions using ReLU networks, emphasizing the dominant role of depth. By leveraging polynomial approximation, Legendre expansions, and complex analysis, the authors construct networks that achieve near-optimal error decay rates, especially when the network depth scales appropriately with the input dimension. The core innovation lies in refined network constructions for powers, multivariate multiplication, and polynomial approximation, which enable the derivation of exponential convergence rates. These results demonstrate that deep architectures can efficiently encode infinitely smooth functions, surpassing traditional polynomial approximation limits. Numerical experiments validate the theoretical findings, showing that increasing depth results in exponential error reduction, a significant improvement over classical methods. The implications are profound: deep networks are not just flexible but fundamentally capable of high-precision approximation for smooth targets, with potential applications in scientific computing, physics simulations, and high-dimensional modeling. While the results rely on certain analyticity assumptions, they open new avenues for understanding the power of depth in neural approximation, guiding future research toward practical algorithms and broader function classes.
Deep Dive
Abstract
In contrast to most studies on neural network approximation theory that characterize results through a single parameter, such as the total number of network parameters, \cite{shen2020deep} pioneered the characterization of approximation rates as a joint function of the width parameter $N$ and the depth parameter $L$, thereby granting greater architectural flexibility. Existing works using the $(N,L)$-characterization focus on function classes with finite smoothness $s$, establishing a typical approximation rate of $\mathcal{O}\left(N^{-2s/d}L^{-2s/d}\right)$ with $d$ denoting the input dimension, which indicates that network depth and width play symmetric roles for these classes. In contrast, this paper establishes upper bounds for the approximation of analytic functions, which possess infinite smoothness, via ReLU networks under the $(N,L)$-characterization. Specifically, we derive approximation rates of $\mathcal{O}\left(N^{-C L^τ}\right)$, where $C>0$ is some constant and $τ>0$ is a parameter influenced by the relation between $L$ and $N$. In particular, $τ=1$ if $N$ scales roughly as $L^d$. Our findings reveal that depth plays a more critical role than width in the context of analytic function approximation. The main technical difficulty of obtaining such upper bounds lies in the trade-off between the smoothness parameters and the approximation accuracy. To overcome this difficulty, we employ refined constructions of several ReLU networks to approximate power functions, multivariate multiplication, and polynomials, which may be of independent interest.