Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning
RFG replaces backpropagation with randomized weight perturbations, reaching 0.0347 Poisson error and an estimated 66% lower training cost.
Key Findings
Methodology
The paper introduces Randomized Forward-Mode Gradient (RFG), which samples a Bernoulli parameter perturbation δθ, computes a directional derivative using one forward pass, and reconstructs a gradient estimate. Spiking neurons use the Integrate-and-Fire model. The nondifferentiable Heaviside derivative is replaced by either a Gaussian surrogate gradient (SG) or a Stein-lemma weak surrogate gradient (WSG). Both global and layer-wise perturbations are evaluated.
Key Results
- For two-dimensional Mexican-hat wavelet regression, SG-RFG-G, SG-RFG-L, and SG-BP obtain relative L2 errors of 0.0426, 0.0342, and 0.0298, respectively. Layer-wise RFG substantially improves over global RFG, although BP remains the most accurate.
- For the 1D Poisson DeepONet, 800 GRF samples are used for training and 800 for testing. SG-BP and WSG-BP achieve 0.0246 and 0.0241 test error, while SG-RFG-G, WSG-RFG-G, SG-RFG-L, and WSG-RFG-L obtain 0.0422, 0.0412, 0.0347, and 0.0359.
- The authors estimate approximately 66% lower training computation because RFG removes the backward pass. On the reaction-diffusion PDE, layer-wise RFG produces predictions similar to BP, whereas adding intermediate local losses does not improve the Poisson experiment.
Significance
The work addresses a persistent mismatch in SNN training. Backpropagation requires chain-rule error transport and stored intermediate activity, limiting biological plausibility and complicating deployment on neuromorphic processors such as Intel Loihi 2. RFG reframes learning as perturbation, observation, and update, while retaining competitive accuracy on scientific regression and PDE operator-learning tasks. This creates a plausible route toward low-power physics modeling, although hardware-level gains remain to be demonstrated.
Technical Contribution
The principal contribution is an integration of randomized forward-mode automatic differentiation with spiking DeepONet and SepONet. The global estimator uses g(θ)≈(∇L·δθ)δθ; the layer-wise variant estimates each layer's directional derivative separately. JVPs compute forward directional information, while SG or WSG handles spike nondifferentiability. Eliminating full backpropagation enables potentially parallel layer-wise perturbations and offers an interface compatible with local synaptic updates and specialized sampling hardware.
Novelty
Unlike ANN-to-SNN conversion and surrogate-gradient backpropagation, this study directly trains SNN scientific operators with randomized weight perturbations and forward-mode gradients. Its novelty lies less in a new neuron model than in operationalizing RFG for spiking MLPs, DeepONet, and SepONet, with controlled comparisons across SG versus WSG, BP versus RFG, and global versus layer-wise perturbations.
Limitations
- The experiments use small networks with one or two hidden layers and widths of 16 or 32. Therefore, stability, convergence, and memory behavior for deep, large-parameter SNNs remain unestablished.
- Only one perturbation direction is used per iteration. The effects of estimator variance, perturbation distribution, learning rate, and multi-direction sampling are not systematically quantified.
Future Work
The authors propose multidirectional perturbations and implementation on Intel Loihi 2. Important extensions include variance-reduced estimators, theoretical convergence and sample-complexity analyses, deeper and higher-dimensional PDEs, and direct measurements of energy, communication, and random-sampling overhead on neuromorphic hardware.
AI Executive Summary
Spiking neural networks process sparse binary events and promise substantial energy savings on neuromorphic processors such as Intel Loihi 2. Yet most directly trained SNNs still rely on backpropagation. Backpropagation requires chain-rule derivatives, cross-layer error transport, and storage of temporal activity—mechanisms that are difficult to reconcile with local biological plasticity and hardware execution.
Wan, Zhang, and Karniadakis propose Randomized Forward-Mode Gradient (RFG). The method perturbs weights randomly, measures the resulting directional change in loss, and estimates updates using forward-mode automatic differentiation rather than a complete backward pass. Integrate-and-Fire neurons generate spikes; Gaussian surrogate gradients or Stein-lemma weak surrogate gradients approximate the derivative of the Heaviside threshold. Global and layer-wise perturbations are tested in spiking MLPs, DeepONet, and SepONet.
Accuracy is competitive but not identical to BP. Mexican-hat regression yields relative L2 errors of 0.0426 for global RFG, 0.0342 for layer-wise RFG, and 0.0298 for BP. In the 1D Poisson problem, layer-wise SG-RFG reaches 0.0347 versus 0.0246 for SG-BP. The paper estimates a 66% reduction in training computation and reports reaction-diffusion predictions from layer-wise RFG that are visually similar to BP. However, experiments are small, single-direction estimates may be noisy, and no Loihi-2 measurements are reported. Multidirectional sampling and hardware validation are therefore decisive next steps.
Deep Analysis
Background
SNNs model membrane dynamics and sparse spikes through LIF or IF neurons. ANN-to-SNN conversion, including approaches such as RMP-SNN and VGG/residual conversion, can preserve accuracy but may lose sparsity benefits. Direct training generally uses surrogate gradients with BP. In scientific machine learning, DeepONet learns mappings between function spaces through branch and trunk networks, while SepONet assigns independent trunk networks to spatial dimensions. These architectures make operator learning a useful testbed for spiking models.
Core Problem
The Heaviside spike function is nondifferentiable, preventing ordinary gradient propagation. Surrogate-gradient BP resolves this numerically but still requires temporal and cross-layer credit assignment, stored activations, and global error signals. Such requirements are biologically implausible and potentially inefficient on neuromorphic hardware. The central question is whether forward-only perturbation measurements can train SNNs for PDE regression without sacrificing too much accuracy.
Innovation
The paper applies RFG to scientific operator learning in SNNs. First, random parameter perturbations estimate directional gradient information instead of performing full reverse-mode differentiation. Second, Gaussian SG is compared with Stein-based WSG. Third, the authors distinguish global perturbation from layer-wise perturbation. Layer-wise RFG isolates each layer's contribution and can be computed in parallel, making it more compatible with local synaptic updates; it also generally outperforms global RFG in the reported tasks.
Methodology
- �� IF dynamics: U_l[t]=U_l[t−1]+W_lS_{l−1}[t]+B_l; a spike S_l=H(U_l−U_threshold) is emitted at threshold and the membrane is reset.
- �� Surrogates: SG uses exp(−x²/(2σ²))/(σ√(2π)); WSG estimates a weak derivative using δ_k∼N(0,σ²I).
- �� RFG: sample Bernoulli δθ, compute the forward directional quantity ∇L·δθ, then estimate g≈(∇L·δθ)δθ.
- �� Layer-wise RFG applies the same construction to each parameter block.
- �� Optimization: θ_l←θ_l−ηg(θ_l).
- �� Architectures: spiking MLPs form the branch and trunk networks of DeepONet and SepONet.
Experiments
The study evaluates Mexican-hat wavelet regression, a 1D Poisson equation, and a nonlinear reaction-diffusion PDE. Function regression uses 10,000 uniformly sampled points on [−1,1]², two hidden layers of width 16, and 32 simulation steps. Poisson uses 800 GRF training and 800 test samples, with one width-32 hidden layer in each branch/trunk net. Reaction-diffusion uses 200 GRF functions, split equally for training and testing, a 100×100 grid, width 16, and 32 steps. SG/WSG, BP/RFG, and global/layer-wise perturbations are compared.
Results
Mexican-hat errors are 0.0426, 0.0342, and 0.0298 for global RFG, layer-wise RFG, and BP. In Poisson, BP errors are about 0.024, while RFG spans 0.0347–0.0422; WSG is not consistently better than SG. Adding local intermediate losses does not help. For reaction-diffusion, a one-hidden-layer SepONet with layer-wise RFG gives predictions comparable to BP. The reported computational estimate favors RFG by approximately 66%.
Applications
Potential uses include low-power PDE surrogate models, event-driven forecasting of dynamical systems, and scientific inference on edge devices. The approach is most attractive when sparse spike computation and local updates matter more than achieving the last increment of accuracy. Practical deployment requires spike encoding, random perturbation generation, optimizer calibration, and benchmarking on Loihi 2 or comparable hardware.
Limitations & Outlook
Evidence is limited to small networks and a few synthetic PDE/operator tasks; there are no large-scale datasets, long-horizon stability studies, or measured chip-level energy results. One-direction perturbations may have high variance, and RFG remains less accurate than BP in Poisson regression. Future work should test multidirectional and variance-reduced estimators, deeper networks, higher-dimensional PDEs, and the overhead of sampling, memory, and communication on actual neuromorphic processors.
Plain Language Accessible to non-experts
Imagine a factory whose workers communicate by flashing lights rather than speaking continuously. A worker flashes only when accumulated pressure crosses a threshold, saving electricity. The factory must learn how to adjust thousands of knobs so that its final product matches a target.
Backpropagation is like a manager who inspects the finished product and walks backward through every department, calculating exactly how much each worker caused the error. This can be accurate, but it requires recording the whole production history and sending instructions through every department. It is not a natural fit for a factory designed around local, independent actions.
RFG uses trial-and-observation. The manager randomly nudges a set of knobs, runs the factory forward once, and checks whether the product improves. If it improves, that nudge is useful; if not, the opposite direction is favored. Checking one department at a time is clearer than changing every department at once, which explains why layer-wise RFG often beats global RFG.
In the paper, this strategy solves wavelet regression and PDE problems. Its errors are slightly higher than backpropagation—0.0342 versus 0.0298 in one test—but estimated training computation falls by about 66%. The idea is promising for low-power chips, though larger factories and real hardware tests are still needed.
ELI14 Explained like you're 14
Think of a game character whose abilities are controlled by many sliders. The character does not communicate with smooth speech; instead, it sends a signal only when its energy bar passes a limit. That signal-based system is a spiking neural network. If the final score is bad, the usual training method rewinds the whole game and calculates how every move affected the ending. Accurate? Often. Simple and brain-like? Not really!
This paper tries a clever shortcut called Randomized Forward-Mode Gradient. It randomly nudges the sliders, plays the game forward, and checks the score. Did the score rise? Keep nudging in that direction. Did it fall? Try the opposite. The method can test all sliders together or inspect one team of sliders at a time. The team-by-team version usually works better because the feedback is less mixed up.
The researchers tested it on a wave-shaped function, a Poisson equation, and a reaction-diffusion equation. For the wave-shaped function, the team-by-team method had error 0.0342, compared with 0.0298 for the traditional method. In the Poisson test, it scored 0.0347 versus about 0.0246. So it is a little less accurate, but the estimated training computation is about 66% lower.
Why care? Future chips may process these signals using very little power—great for robots, sensors, or portable science tools. But the study used small networks and only one random trial direction each round. Next, researchers want to try several directions at once and test everything on a Loihi 2 chip. That is where the real boss battle begins!
Glossary
Spiking Neural Network
A neural network that communicates through discrete spike events rather than continuous activations. Sparsity can reduce energy use.
All MLP, DeepONet, and SepONet components in the paper are spiking networks.
Randomized Forward-Mode Gradient
A gradient estimator based on random parameter directions and forward directional derivatives. Its central estimate is approximately (∇L·δθ)δθ.
It replaces full backpropagation in the proposed SNN training procedure.
Surrogate Gradient
A smooth approximation to the derivative of a threshold spike function. It allows optimization despite the nondifferentiable Heaviside operation.
The paper compares Gaussian SG with Stein-based WSG.
DeepONet
An operator-learning architecture with branch and trunk networks. Their feature combination approximates mappings from input functions to output functions.
The spiking DeepONet solves the 1D Poisson operator-learning task.
SepONet
A separable operator network using independent trunk networks for different spatial dimensions. Their outer-product representation is combined with branch features.
A spiking SepONet is used for the reaction-diffusion PDE.
Jacobian-Vector Product
The product of a Jacobian with a chosen direction, giving a directional derivative without materializing the full Jacobian. It is the computational primitive of forward-mode AD.
RFG uses JVP-like forward calculations inside the perturbation estimator.
Open Questions Unanswered questions from this research
- 1 The convergence and sample complexity of RFG for deep, large SNNs remain unknown. Theory must connect perturbation variance, learning rate, network depth, and spike sparsity.
- 2 Single-direction Bernoulli perturbations may be noisy. It is unresolved whether parallel multidirectional sampling can reduce variance enough to improve both accuracy and hardware efficiency.
- 3 The paper reports no Loihi-2 energy measurements. Operation-count reductions may be offset by random sampling, communication, or memory overhead.
Applications
Immediate Applications
Edge PDE surrogate modeling
Engineering teams could train spiking DeepONet models for parameterized PDEs and deploy them on low-power edge devices. Small operator-learning problems and prior validation against BP are currently important prerequisites.
Neuromorphic dynamical forecasting
Event-driven sensors could use spiking operators to predict diffusion, reaction, or structural responses. RFG is useful when local updates and forward-only hardware execution are more important than maximum numerical accuracy.
Long-term Vision
Scientific computing on Loihi 2
If multidirectional RFG reduces estimator variance and maps efficiently to Loihi 2, it could enable low-energy training and inference for physical simulation, robotics, digital twins, and embedded environmental models.
Abstract
Spiking neural networks (SNNs) represent a promising approach in machine learning, combining the hierarchical learning capabilities of deep neural networks with the energy efficiency of spike-based computations. Traditional end-to-end training of SNNs is often based on back-propagation, where weight updates are derived from gradients computed through the chain rule. However, this method encounters challenges due to its limited biological plausibility and inefficiencies on neuromorphic hardware. In this study, we introduce an alternative training approach for SNNs. Instead of using back-propagation, we leverage weight perturbation methods within a forward-mode gradient framework. Specifically, we perturb the weight matrix with a small noise term and estimate gradients by observing the changes in the network output. Experimental results on regression tasks, including solving various PDEs, show that our approach achieves competitive accuracy, suggesting its suitability for neuromorphic systems and potential hardware compatibility.