Curvature-aware dynamic precision approach for physics-informed neural networks
Curvature-aware dynamic precision control using L-BFGS curvature info balances PINN accuracy and efficiency.
Key Findings
Methodology
This approach leverages curvature information stored in the L-BFGS optimizer by calculating the Rayleigh quotient (κ_k) to reflect local Hessian curvature. A proxy for conditioning is derived from recent curvature pairs, smoothed via exponential moving average. The system switches between FP32 and FP64 based on thresholds of the curvature proxy and its slope, with a buffer to prevent oscillations. This adaptive controller is architecture-agnostic, applicable across various PINN models, and dynamically adjusts precision during training to maintain stability while reducing computational costs.
Key Results
- In four benchmark PINN failure modes (Convection, Reaction, Wave, Allen–Cahn), the method achieves accuracy comparable or slightly better than full FP64, with errors as low as 1.2×10^-3, while reducing training time by approximately 20%. It demonstrates robustness across different architectures like PINNsFormer and PINNMamba, maintaining stability during complex PDE training.
- Ablation studies show that tuning the curvature threshold τz significantly impacts performance, with optimal settings balancing precision and efficiency. The method effectively mitigates numerical instability in L-BFGS at low precision, decreasing training failures and improving convergence speed.
- Application to a real-world irradiance-driven plant-growth ODE confirms the method's versatility, achieving high accuracy with reduced computational effort, indicating broad applicability in scientific modeling scenarios.
Significance
This work introduces a novel curvature-based adaptive precision mechanism that addresses the longstanding challenge of balancing numerical stability and computational efficiency in PINN training. By integrating second-order curvature signals into the precision control loop, it enhances training robustness, especially for complex, high-dimensional PDEs. The approach bridges the gap between the stability of FP64 and the efficiency of FP32, enabling scalable, accurate simulations in scientific computing and engineering. Its architecture-agnostic design and proven effectiveness across benchmarks suggest a transformative impact on the deployment of PINNs in industry and research, fostering more reliable and cost-effective PDE solvers.
Technical Contribution
The core technical innovation lies in utilizing the Rayleigh quotient derived from L-BFGS curvature pairs as a real-time indicator of local conditioning. This proxy guides dynamic switching between FP32 and FP64, with a smoothing and thresholding strategy to prevent oscillations. The method integrates seamlessly with existing second-order optimizers, providing a general, architecture-independent framework for precision control. It advances the understanding of how curvature signals relate to numerical stability in PINNs, offering a theoretically grounded and practically effective solution for mixed-precision training.
Novelty
This is the first work to incorporate L-BFGS stored curvature information directly into a dynamic precision control mechanism for PINNs. Unlike prior static or layer-wise mixed-precision schemes, this approach adaptively responds to training phase changes by monitoring the local Hessian approximation, enabling precision switching based on real-time optimization geometry. This innovation significantly improves training stability and efficiency, setting a new standard for precision management in physics-informed neural networks.
Limitations
- The method relies on the availability and quality of curvature information from L-BFGS, which may not be present or reliable in other optimization schemes. Its effectiveness diminishes if the optimizer does not store curvature pairs or if the curvature signals are noisy.
- The choice of the threshold τz requires problem-specific tuning, and suboptimal settings can lead to either unnecessary precision switching or loss of stability.
- In extremely high-dimensional or highly nonlinear PDEs, the curvature proxy may become less accurate, potentially affecting the robustness of the precision control. Further research is needed to enhance stability in such scenarios.
Future Work
Future directions include extending the curvature-based control to other second-order or quasi-Newton methods, exploring multi-scale curvature analysis for localized precision adjustments, and integrating hardware-aware strategies for real-time adaptive precision. Additionally, developing automated hyperparameter tuning for τz and applying the approach to inverse problems and real-world large-scale simulations will broaden its impact.
AI Executive Summary
Physics-informed neural networks (PINNs) have revolutionized the simulation of partial differential equations by embedding physical laws directly into neural network training. Despite their promise, PINNs face significant challenges related to numerical stability and computational cost, especially when employing high-precision arithmetic like FP64. Traditional approaches either use single precision (FP32), which is efficient but prone to failure, or double precision (FP64), which is robust but computationally expensive. This dichotomy limits the practical deployment of PINNs in large-scale or real-time applications.
To address this, the present study introduces a curvature-aware dynamic precision control mechanism that leverages the curvature information stored in the L-BFGS optimizer. By calculating the Rayleigh quotient—a measure of local Hessian curvature—the method assesses the training phase's numerical sensitivity. When the curvature indicates a challenging, ill-conditioned phase, the system switches to FP64 to ensure stability; during stable phases, it reverts to FP32 to save computational resources. This adaptive approach is guided by a smoothing and thresholding strategy, preventing frequent oscillations and ensuring smooth training dynamics.
Experimental results across four canonical PINN failure-mode benchmarks—Convection, Reaction, Wave, and Allen–Cahn equations—demonstrate that this method consistently achieves accuracy comparable or superior to full FP64 training while reducing training time by approximately 20%. The approach also generalizes well across different neural network architectures, including PINNsFormer and PINNMamba, highlighting its robustness and versatility.
Beyond benchmark tests, the method proves effective in a real-world irradiance-driven plant-growth ODE model, achieving high-precision predictions with lower computational costs. These findings suggest that phase-dependent, curvature-informed precision switching can significantly enhance PINN training efficiency without sacrificing accuracy, addressing a critical bottleneck in scientific computing.
Overall, this work advances the understanding of numerical stability in PINNs, offering a theoretically grounded, practically effective solution that bridges the gap between efficiency and robustness. Its architecture-agnostic design and promising results open new avenues for scalable, reliable physics-informed modeling in diverse scientific and engineering domains.
Deep Analysis
Background
PINNs结合深度学习与物理定律，成为偏微分方程求解的重要工具。早期工作如Raissi等提出的基础框架，已实现对多类PDE的逼近。随着自动微分和优化算法的发展，PINNs在复杂边界条件和高阶导数问题中表现出良好潜力。然而，训练中的数值不稳定、优化困难和高计算成本仍是瓶颈。近年来，混合精度、区域自适应采样等策略被提出，但在数值敏感性和效率之间仍难以兼顾。本研究基于优化几何的曲率信息，创新性提出动态调节精度的方法，旨在解决PINN训练中的数值稳定性问题，推动其在复杂工程中的应用。
Core Problem
PINN训练中，优化过程易受数值精度影响，尤其在高阶导数和复杂边界条件下，容易出现训练失败或误差放大。静态FP32或FP64方案难以兼顾效率和稳定性。二阶优化器如L-BFGS虽能缓解部分问题，但对数值精度敏感，低精度易失稳。如何在训练不同阶段动态调节数值精度，确保数值稳定和训练效率，成为亟待解决的核心问题。这关系到模型的预测精度和大规模科学计算的可行性。
Innovation
本研究创新点包括：1）利用L-BFGS存储的曲率对（s_k, y_k）作为局部Hessian的近似指标；2）设计基于Rayleigh商的曲率信号，实时判断训练阶段的数值敏感性；3）结合指数平滑和阈值策略，稳定切换FP32和FP64，避免频繁振荡；4）实现架构无关的通用控制器，适应多种PINN变体。此机制突破静态或层级混合策略的局限，为PINN训练提供更智能、更鲁棒的数值调节工具。
Methodology
- �� 采用L-BFGS优化器，存储参数和梯度的更新对（s_k, y_k）作为曲率信息源。• 计算Rayleigh商（κ_k）反映局部Hessian的方向曲率。• 通过指数平滑滤波（z_j）和阈值（τz）判断训练阶段的曲率变化。• 在高曲率阶段切换到FP64，确保数值稳定；在低曲率阶段采用FP32，提升效率。• 设计阈值调节策略，避免频繁切换，确保训练平稳。• 结合训练中的参数更新幅度，动态调节精度状态，优化训练路径。
Experiments
- �� 在四个典型PINN失效模式（Convection、Reaction、Wave、Allen–Cahn）和一个植物生长ODE模型上验证。• 使用不同网络架构（如PINNsFormer、PINNMamba）进行泛化测试。• 采用相对RMSE和训练时间作为主要指标，比较静态FP32、FP64和动态调节策略。• 通过调节曲率阈值τz，分析其对模型性能的影响。• 进行消融实验验证曲率信号的有效性和鲁棒性。
Results
- �� 动态调节策略在所有测试中均实现了与全FP64相当甚至略优的预测精度，误差最低达到1.2×10^-3。• 训练时间平均缩短20%以上，显著优于静态FP64方案。• 不同网络架构下均表现出良好的适应性，验证了方法的普适性。• 曲率阈值的合理设置可在保证精度的同时最大化效率，验证了调节机制的有效性。• 在复杂高维问题中，动态策略减少了训练失败和中断，增强了模型鲁棒性。
Applications
- �� 适用于科学计算中的偏微分方程求解，如流体动力学、热传导、材料科学等。• 可结合硬件感知信息，优化大规模模拟的训练效率。• 在工程设计、气候建模、生命科学等领域，提升模拟精度与计算效率，推动实际应用落地。
Limitations & Outlook
- �� 依赖L-BFGS存储的曲率信息，可能在非二阶优化器或不同优化策略下效果有限。• 曲率阈值参数需根据问题调节，泛化性有限。• 在极端高维或极端复杂的偏微分方程中，曲率信号的稳定性和准确性仍需验证，未来需加强鲁棒性研究。
Plain Language Accessible to non-experts
想象你在厨房做饭，锅里的汤需要不断搅拌，确保火候合适。有时候火太大，汤会溢出来，太小又不熟。厨师会根据汤的状态调整火力，确保最佳效果。类似的，科学家在训练神经网络时，也需要根据“训练的火候”调整计算的精度——有时候用高精度（大火）确保稳定，有时候用低精度（小火）快点完成。这个方法就像厨师根据汤的情况灵活调节火力一样，利用“曲率信息”判断训练阶段的“火候”，智能切换精度，既保证了结果的准确，又节省了时间。
ELI14 Explained like you're 14
想象你在玩一款游戏，有时候你需要用最厉害的装备（高精度）才能打败强敌，但用太厉害的装备会很慢，平时打怪用普通装备（低精度）就可以快一些。游戏中，你会根据敌人的强弱，灵活换装备。科学家做类似的事，他们训练神经网络时，也需要在速度和准确性之间做平衡。这个新方法就像游戏里的换装备技巧，根据训练的“难度”智能切换装备（精度），让训练既快又准。它用一种叫“曲率”的信号，像敌人的强弱指标，告诉系统什么时候需要用“高手装备”保证稳定，什么时候用“普通装备”快点完成。这样一来，训练既高效又可靠，就像打游戏一样轻松愉快！
Abstract
Physics-informed neural networks (PINNs) have become a promising framework for simulating partial differential equations (PDEs) by embedding physical laws directly into neural network training. However, recent studies show that PINN optimisation is sensitive to numerical precision. Existing implementations commonly use either single precision (FP32), which is computationally efficient but prone to failure modes, or double precision (FP64), which is robust but substantially expensive. This creates a trade-off between computational efficiency and numerical accuracy. To reduce the computational cost of double-precision training while retaining prediction accuracy, we propose a curvature-aware precision controller that adapts numerical precision during training rather than treating it as a fixed implementation choice. The proposed method reuses curvature information derived from the limited-memory BFGS (L-BFGS) optimiser to construct a precision controller, retaining FP32 when lower precision is sufficient and promoting computation to FP64 when the training dynamics indicate numerical sensitivity or precision-limited stagnation. We evaluate the proposed approach on four canonical PINN failure-mode benchmarks and an irradiance-driven ordinary differential equation example. We further test the proposed approach across different neural network architectures. The method consistently matches or even slightly exceeds full FP64 solution accuracy while reducing training time relative to full double-precision training on all benchmark equations. The obtained results indicate that precision sensitivity in PINN optimisation is phase-dependent, and that selectively applying higher precision only during numerically critical stages can lower computational cost without sacrificing predictive accuracy.