Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms
Theorem 3 proves GELU-like activations achieve Sobolev super-convergence at error rate (NL)^−2(n−m)/d.
Key Findings
Methodology
The paper directly constructs fully connected σ-networks in W^{m,∞}, rather than approximating a ReLU network and then replacing ReLU by a smooth activation. The proof combines the global m-th-order quasi-decay condition, local nonlinear behavior, fixed-size subnetworks for x², multiplication and monomial modules, smooth partitions of unity, bit-extraction ideas, and the Bramble–Hilbert lemma.
Key Results
- Theorem 3 states that for f∈W^{n,∞}([0,1]^d), m<n, there exists a σ-network with depth C1L log L and width C2N log N satisfying ||f−φ||_{W^{m,∞}}≤C0||f||_{W^{n,∞}}N^{-2(n−m)/d}L^{-2(n−m)/d}.
- Classical finite-element and spectral approximations typically achieve M^{-(n−m)/d} with M degrees of freedom. Since the network parameter count is O(N²L) up to logarithms, fixed width gives an L^{-2(n−m)/d} depth rate, theoretically faster than continuous approximation.
- No independent datasets, training runs, or percentage improvements are reported. The contribution is theoretical; Table 1 covers GELU, Tanh, Sigmoid, Softplus, SiLU, Mish, arctan, and others, with 0≤m≤n−1 for many smooth activations.
Significance
The work addresses a central weakness in neural PDE analysis: accurate function values do not automatically imply accurate derivatives. Deep Ritz energy error is controlled by an H1-type error, while PINN residuals can be controlled through H2-type regularity. Consequently, a unified Sobolev approximation theorem directly informs PDE error analysis. The result also gives a mathematical explanation for why depth may outperform standard continuous discretizations at the representation level.
Technical Contribution
Condition 1 requires |σ^{(k)}(x)−H^{(k)}(x)|≤min{C|x|^{k+1},G} for k≤m, while Condition 2 requires local smooth non-affinity, σ''(a)≠0. A depth-one, width-four σ-network extracts x² through scaled finite differences. Multiplication, monomials, polynomial approximation, and smooth partitions then yield the global construction. This extends ReLU-based Sobolev super-convergence to single smooth activations.
Novelty
Earlier super-convergence results were concentrated on ReLU, powers of ReLU, or mixed activations. Smooth GELU cannot uniformly approximate the nondifferentiable ReLU in W^{1,∞}, so a simple composition argument fails. This paper bypasses that obstruction by constructing the target-activation network directly and supplying practical sufficient conditions that cover many popular activations.
Limitations
- The paper provides an existence and approximation theorem, not a trainability theorem. It does not show that stochastic gradient descent can discover the constructed weights or reproduce the asymptotic rate.
- The proof notes that parameters may diverge as ε, δ→0 or M→∞, potentially causing poor conditioning and optimization instability.
- Condition 1 excludes ReLU^k for k≥2, which requires the separate approximation properties (i)–(ii); constants and practical parameter magnitudes are not optimized.
Future Work
Future work should bound weight norms, characterize optimization reachability, and separate representation, sampling, discretization, optimization, and generalization errors. Important directions include irregular domains, limited regularity, adaptive architectures, high-dimensional PDEs, and controlled empirical comparisons among GELU, Tanh, SiLU, and ReLU. The key challenge is converting existential constructions into stable, trainable scientific-computing systems.
AI Executive Summary
Neural PDE solvers must approximate both a solution and its derivatives. This is harder than ordinary regression: a network may fit u accurately while producing a poor gradient or Hessian. Classical finite-element and spectral schemes typically achieve M^{-(n−m)/d} in Sobolev norms. ReLU networks can sometimes improve this rate through depth, but their piecewise-linear derivative is poorly suited to smooth derivative fields. A second problem is that a smooth activation such as GELU cannot approximate the kink of ReLU uniformly in W^{1,∞}.
Yang and He propose a direct construction for a single general activation σ. Under a global quasi-decay condition and a local nonlinear condition, a fixed-size subnet approximates x²; multiplication modules then build monomials and polynomials. Smooth partitions of unity, bit-extraction constructions, and the Bramble–Hilbert lemma assemble these local approximations into a global network. The strategy avoids the invalid “ReLU first, smooth replacement later” argument and applies to GELU, Tanh, Sigmoid, Softplus, SiLU, Mish, and related functions.
Theorem 3 gives depth C1L log L and width C2N log N, with W^{m,∞} error C0||f||_{W^{n,∞}}(NL)^{-2(n−m)/d}. Because the parameter count is O(N²L) up to logarithmic factors, fixed width yields a depth rate twice the exponent of the standard continuous rate. The paper reports no datasets, training experiments, or numerical PDE benchmarks; its achievement is a constructive approximation theorem. It therefore supplies a rigorous foundation for PINNs and Deep Ritz methods, while leaving trainability, parameter conditioning, and real-world computational cost open.
Deep Analysis
Background
PINNs, Deep Ritz, and related methods use neural networks to represent PDE solutions and their derivatives. Prior Sobolev results either matched classical approximation rates or relied heavily on ReLU and ReLU^k. ReLU can approximate a smooth solution well while producing visibly inaccurate derivatives, as illustrated by the one-dimensional Poisson example. The unresolved question is whether common single smooth activations can retain deep-network super-convergence.
Core Problem
The natural reduction from a smooth activation to ReLU fails in W^{1,∞}: GELU(Kx)/K converges uniformly to ReLU but not uniformly in derivative norm because ReLU is nondifferentiable at zero. The composition estimate in equation (3) explicitly requires a W^{1,∞} outer error. Therefore, even a final W^{1,p} target cannot be obtained by that route.
Innovation
The paper introduces two sufficient frameworks. Condition 1 controls the activation and its derivatives relative to the Heaviside function at large inputs; Condition 2 supplies local curvature through σ''(a)≠0. Unlike ReLU-only theories, the proof directly constructs σ-networks. Unlike L∞-only general-activation results, it establishes W^{m,∞} rates and thereby supports derivative-sensitive PDE analysis.
Methodology
- �� Lemma 7 uses a depth-one, width-four σ-network and scaled finite differences to cancel constant and linear Taylor terms, producing x².
- �� Multiplication modules construct xy, x^p, products x1…xd, and monomials x^α; the basic product architecture has depth d−1 and width 8d−4, with a logarithmic-depth alternative.
- �� ReLU^m-inspired subnetworks and bit-extraction mechanisms create smooth local partitions of unity.
- �� On each partition cell, a local polynomial approximates f through the Bramble–Hilbert lemma.
- �� Error accumulation and network-size accounting yield depth C1L log L, width C2N log N, and the rate in equation (5).
Experiments
This is a theoretical paper and reports no MNIST, CIFAR-10, PDE benchmark, optimizer, training curve, or ablation study. Figure 1 uses a one-dimensional Poisson problem to illustrate that a deep ReLU network may fit u while mismatching u′; it is motivation rather than a controlled experiment. The theorem assumes Ω=[0,1]^d, f∈W^{n,∞}, and supports W^{m,p} interpretation for 1≤p≤∞.
Results
Equation (5) gives error C0||f||_{W^{n,∞}}N^{-2(n−m)/d}L^{-2(n−m)/d}. Classical continuous methods give M^{-(n−m)/d}; with parameter count O(N²L), fixed width produces L^{-2(n−m)/d}. Table 1 reports 0≤m≤n−1 for Sigmoid, Tanh, arctan, dSiLU, SRS, Softplus, SiLU, Mish, and GELU. ReLU-family cases have separate, more restricted statements.
Applications
The theorem can guide activation and architecture choices for PINNs, where differential residuals depend on derivative approximation, and Deep Ritz methods, where energy error is tied to H1 error. Potential targets include elliptic boundary-value problems and other derivative-sensitive scientific models. Practical deployment additionally requires boundary enforcement, quadrature, sampling, optimization, and stability analysis.
Limitations & Outlook
The result concerns representational existence, not an end-to-end solver guarantee. Constructed weights may become large as ε and δ shrink, and the constants are not optimized for implementation. The assumptions may be difficult for irregular domains, nonsmooth solutions, or high-dimensional problems. No empirical comparison validates the predicted rate. Future work must combine approximation theory with optimization, discretization, sampling, conditioning, and generalization analyses.
Plain Language Accessible to non-experts
Imagine a factory producing a detailed landscape map. The goal is not merely to match the height of every hill; the map must also show correct steepness, because vehicles need to know how hard each slope is to climb. A traditional factory cuts the map into many small squares and places one simple pattern in each square. More squares improve accuracy, but only at a standard pace.
This paper designs a deep assembly line. Early stations learn basic operations such as squaring and multiplication. Later stations combine these pieces into local polynomial patterns. A final set of smooth blending stations joins neighboring patterns without visible seams. Crucially, the factory can use smooth tools such as GELU or Tanh instead of relying on the sharp corner of ReLU. The authors prove that, under specific shape conditions, this smooth factory can match both heights and slopes.
The mathematical blueprint predicts an error decline of roughly (NL)^−2(n−m)/d, faster than the usual continuous-factory rate. But it is a blueprint, not a production audit: the paper does not demonstrate that a practical training algorithm can find the required settings, nor that large internal parameters are easy to operate.
ELI14 Explained like you're 14
Think of a game where you must build a mountain from blocks. Getting the mountain’s height right is only half the job; the slopes must also feel natural when a character runs across them. ReLU blocks are easy to stack, but their sharp corners can make the slope suddenly change. That is bad if your game needs smooth motion.
This paper asks whether smoother blocks—like GELU, Tanh, or Mish—can build both the mountain and its slopes accurately. The authors create a multi-floor workshop. Some floors learn how to square numbers and multiply them. Other floors combine these simple pieces into many small landscape patches. The final floors blend the patches so the whole mountain looks continuous.
The theorem says the error can shrink like (NL)^−2(n−m)/d. N is roughly the number of workers on each floor, and L is the number of floors. Adding floors can therefore help much more than simply adding flat blocks. The paper shows this idea applies to many popular activation functions.
But there is an important twist: this is not a leaderboard from an actual game. There are no training scores, datasets, or full PDE demonstrations. The authors prove that a perfect workshop exists; the next challenge is teaching a real optimizer to build it without using enormous or unstable settings!
Glossary
Sobolev space
A function space that measures both a function and its weak derivatives. W^{n,p} requires derivatives through order n to belong to L^p.
The target lies in W^{n,∞}, while error is measured in W^{m,p}.
Super-convergence
A rate faster than the standard optimal rate of continuous approximation. Here the central bound is (NL)^{-2(n−m)/d}.
It compares deep networks with finite-element and spectral methods.
Quasi-decay condition
A condition making σ and its derivatives approximate Heaviside-related behavior away from zero. It enables accurate switching and localization modules.
Condition 1 is the global activation requirement.
Bramble–Hilbert lemma
A local approximation result bounding the error of polynomial projection in terms of function regularity and cell size.
It completes the local-to-global approximation proof.
Deep Ritz method
A neural PDE method that minimizes a variational energy rather than a pointwise residual. Its approximation error is closely related to an H1 norm.
The paper uses it to motivate Sobolev error theory.
Open Questions Unanswered questions from this research
- 1 The construction may require diverging parameters as ε, δ→0. It remains unclear how weight magnitude affects conditioning, optimization stability, generalization, and the ability of gradient methods to reach the theoretical network.
- 2 The theorem’s behavior on irregular domains, nonsmooth solutions, and very high-dimensional PDEs is unresolved. A complete theory must also include sampling, quadrature, boundary, and optimization errors.
Applications
Immediate Applications
Derivative-aware PINNs
PDE researchers can use GELU, Tanh, or SiLU when residuals require accurate derivatives, while using the theorem to justify Sobolev-oriented architecture choices. The target must have sufficient regularity, and training and sampling errors must be monitored separately.
Deep Ritz elliptic solvers
Smooth activations satisfying the stated conditions may improve gradient-field representation in energy minimization. Implementations still need reliable Neumann constraints, zero-mean handling, quadrature, and optimization diagnostics.
Long-term Vision
Certified neural scientific computing
A future system could combine approximation, discretization, optimization, and generalization bounds to produce predictable neural PDE solvers, with fair cost comparisons against adaptive finite elements.
Abstract
This paper establishes a comprehensive approximation result for deep fully-connected neural networks with commonly-used and general activation functions in Sobolev spaces $W^{n,\infty}$, with errors measured in the $W^{m,p}$-norm for $m < n$ and $1\le p \le \infty$. The derived rates surpass those of classical numerical approximation techniques, such as finite element and spectral methods, exhibiting a phenomenon we refer to as \emph{super-convergence}. Our analysis shows that deep networks with general activations can approximate weak solutions of partial differential equations (PDEs) with superior accuracy compared to traditional numerical methods at the approximation level. Furthermore, this work closes a significant gap in the error-estimation theory for neural-network-based approaches to PDEs, offering a unified theoretical foundation for their use in scientific computing.