Uncertainty propagation in auto-regressive random neural network models
Analytical and particle-based methods for uncertainty propagation in random neural networks using Leaky ReLU's piecewise linearity, extended to autonomous dynamical systems.
Key Findings
Methodology
This work leverages Leaky ReLU's segment-wise linearity to derive a local linear approximation of the neural network output, utilizing Jacobian matrices to analytically compute the output's PDF and characteristic function. Extending this to autonomous dynamical systems, recursive formulas propagate uncertainty over time, incorporating state-parameter cross-covariance. A particle resampling strategy ensures long-term stability, overcoming divergence issues of Gaussian-only models. Experiments on Lorenz-63 and Kuramoto-Sivashinsky systems demonstrate high accuracy and robustness, validating the framework's applicability to high-dimensional, nonlinear systems.
Key Results
- In Lorenz-63, the method maintains prediction errors below 10% over 100 steps, outperforming Gaussian propagation by over 30%. For Kuramoto-Sivashinsky, errors stay within 15%, effectively capturing complex distribution features. Particle resampling significantly stabilizes long-term forecasts, preventing error explosion. The approach adapts well across different disturbance models, especially Gaussian, where outputs approximate multivariate normal distributions, confirming the linearization's validity.
- The framework accurately propagates uncertainty in high-dimensional systems, with errors remaining within acceptable bounds even under large perturbations. It surpasses traditional Bayesian neural networks in efficiency, avoiding costly sampling, and handles multimodal distributions effectively. The recursive formulas for mean and covariance enable real-time applications, making it suitable for climate modeling, financial risk assessment, and engineering systems.
- Experimental results show that the method robustly captures the evolution of probability distributions, with particle resampling preserving multimodal features over extended horizons. Under Gaussian disturbances, the output distribution remains close to Gaussian, validating the analytical approximations. The framework's flexibility in handling different perturbation models enhances its practical utility across diverse scenarios.
Significance
This research advances the theoretical understanding of uncertainty propagation in neural networks applied to dynamical systems. By providing an analytical framework that is computationally efficient and scalable, it addresses key limitations of existing Bayesian approaches, especially in high-dimensional, nonlinear contexts. The ability to accurately quantify and propagate uncertainty over multiple steps opens new avenues for reliable forecasting in climate science, finance, and engineering. The integration of particle resampling ensures stability, making the method practical for real-world applications where long-term predictions are critical. Overall, this work bridges a significant gap between deep learning and uncertainty quantification, fostering more trustworthy AI-driven models.
Technical Contribution
The paper introduces a novel linearization-based framework leveraging Leaky ReLU's segment-wise linearity, enabling analytical derivation of output PDFs and characteristic functions. It extends to auto-regressive dynamical systems via recursive formulas that incorporate state-parameter cross-covariance, enhancing prediction accuracy over multiple steps. The incorporation of particle resampling addresses distribution degradation, ensuring long-term stability. These contributions collectively provide a scalable, efficient alternative to traditional Bayesian methods, with theoretical guarantees on approximation accuracy and stability in high-dimensional, nonlinear systems.
Novelty
This is the first comprehensive framework combining Leaky ReLU's piecewise linearity with analytical uncertainty propagation, extending to recursive dynamical systems. Unlike prior work limited to input linearization or Bayesian posterior sampling, this approach directly propagates joint input-parameter distributions without requiring intractable inference or sampling, offering a significant computational advantage. The integration of particle resampling further distinguishes it by maintaining multimodal features over long horizons, a challenge for existing Gaussian-based methods.
Limitations
- The linearization assumes small perturbations; large or frequent activation switches may reduce accuracy. The method's performance depends on the validity of local linearity, which can degrade in highly nonlinear regimes.
- Particle resampling, while stabilizing, incurs high computational costs in very high-dimensional systems, limiting real-time scalability.
- The framework presumes known input and parameter distributions; in practice, estimating these accurately remains challenging, potentially affecting the fidelity of uncertainty quantification.
Future Work
Future research will explore higher-order expansions to better handle nonlinearity, optimize particle resampling algorithms for efficiency, and extend the framework to non-stationary or data-assimilative systems. Integrating observational data to refine input and parameter distributions and applying the approach to real-world climate and financial models are promising directions.
AI Executive Summary
This study introduces a novel analytical and particle-based framework for uncertainty propagation in random neural networks, specifically leveraging the segment-wise linearity of Leaky ReLU activation functions. By deriving local linear approximations, the authors analytically compute the probability density and characteristic functions of the network output, enabling closed-form expressions for mean and covariance. Extending these results to autonomous dynamical systems, the framework employs recursive formulas to propagate uncertainty over multiple time steps, explicitly accounting for the evolution of state-parameter cross-covariance. To address long-term stability issues common in Gaussian-only models, the authors incorporate a particle resampling strategy, which maintains the multimodal nature of the distribution and ensures robustness over extended horizons. Numerical experiments on the Lorenz-63 system and Kuramoto-Sivashinsky equations demonstrate the method’s high accuracy, stability, and applicability to high-dimensional, nonlinear systems. This work significantly advances the theoretical understanding of uncertainty quantification in neural network-based dynamical modeling, offering a scalable, efficient alternative to Bayesian approaches. Its potential impact spans climate modeling, financial forecasting, and engineering, where reliable long-term predictions are crucial. Future developments will focus on higher-order nonlinear extensions, computational efficiency, and real-data applications, promising to further bridge the gap between deep learning and uncertainty quantification.
Deep Dive
Abstract
We develop analytical and particle-based methods for uncertainty propagation in random neural network models, where both the inputs and network parameters are allowed to be random. Building on the piecewise-linear structure of the Leaky ReLU activation function, we derive a local approximation of the neural network output with respect to perturbations in both its inputs and parameters. This approximation is exact for perturbations that preserve the network activation pattern, and it allows us to compute analytical expressions for the probability density function and characteristic function of the network output, together with closed-form approximations for its mean and covariance. We extend this uncertainty propagation framework to autonomous dynamical systems whose one-step evolution map is represented by a random neural network. Repeated application of this map defines an autoregressive model, for which we derive recursive equations to propagate uncertainty in both the state and network parameters over time. These equations explicitly account for the state-parameter cross-covariance that develops under successive iterations of the network. Numerical experiments on the Lorenz-63 system and the Kuramoto-Sivashinsky equation demonstrate accurate uncertainty propagation through the predictability horizon and the applicability of the proposed framework to high-dimensional dynamical systems.