Overtone: Cyclic Patch Modulation for Clean, Efficient, and Flexible Physics Emulators
Overtone employs cyclic patch size modulation via CSM and CKM modules, reducing spectral harmonic errors by up to 40%, enabling flexible, compute-adaptive PDE surrogates.
Key Findings
Methodology
Overtone introduces two architecture-agnostic modules—CSM (dynamic stride modulation) and CKM (kernel resizing)—to enable cyclic patch size adjustments during autoregressive PDE prediction. By alternating patch scales (e.g., 4, 8, 16), the framework disperses harmonic spectral errors that typically accumulate at fixed scales. This process involves training with random patch sizes and during inference, cyclically adjusting patch parameters to prevent error reinforcement at specific frequencies. The modules operate independently of backbone architecture, compatible with vanilla and axial ViT models. Empirical results across 2D and 3D PDE benchmarks demonstrate up to 40% reduction in VRMSE, with models maintaining or surpassing fixed-patch baselines across various compute budgets, without retraining.
Key Results
- The cyclic modulation strategy consistently reduces long-term spectral errors, achieving up to 40% lower VRMSE compared to fixed patch models. In multiple PDE tasks, a single flexible model outperforms fixed-patch counterparts trained separately at different scales, across diverse inference compute budgets.
- Spectral analysis reveals that alternating patch sizes prevents coherent error reinforcement at harmonic frequencies, leading to more stable long-horizon predictions. The approach is architecture-agnostic, working with both vanilla and axial ViT backbones.
- In practical scenarios, the models dynamically trade accuracy for computational efficiency, enabling resource-aware deployment. Experimental data show that reducing patch size from 16 to 4 triples inference time but improves VRMSE by approximately 2×, demonstrating effective compute-accuracy balancing.
Significance
This work addresses fundamental limitations of fixed-patch PDE surrogates by introducing a novel spectral error dispersal mechanism. It enhances long-term stability and broadens the applicability of deep learning models in physics-based simulations. The test-time adjustable framework significantly advances the state-of-the-art, offering a practical solution for resource-constrained environments and real-time applications, thus bridging the gap between high-fidelity physics simulation and scalable AI models.
Technical Contribution
The core technical innovation lies in the cyclic patch modulation strategy, implemented via CSM and CKM modules, which decouple patch size from architecture constraints. These modules enable dynamic, test-time adjustment of tokenization scales, effectively mitigating harmonic spectral errors. The approach leverages spectral analysis insights, integrating them into a training-free, inference-time resource tuning mechanism. Extensive experiments validate that this method outperforms fixed-patch models in accuracy, stability, and computational flexibility, establishing a new paradigm for PDE surrogate modeling.
Novelty
This is the first work to systematically incorporate cyclic patch size modulation during autoregressive PDE prediction, explicitly targeting spectral harmonic error dispersion. Unlike prior fixed-patch models or static adaptive methods, the proposed approach dynamically varies tokenization scales during inference, providing a novel means to control spectral error distribution without retraining. This innovation fundamentally shifts the paradigm of patch-based surrogate modeling, enabling compute-aware, stable long-term predictions.
Limitations
- The effectiveness of cyclic modulation depends on appropriate parameter choices; poorly tuned schedules may reduce benefits or introduce artifacts. The method's performance may diminish in scenarios with highly complex or high-frequency features, requiring further adaptation.
- While flexible, the approach adds complexity to inference scheduling and may require empirical tuning for optimal results. Its scalability to extremely high-dimensional or highly nonlinear PDEs remains to be fully validated.
- The current framework assumes spectral errors dominate; in some applications, spatial errors or other artifacts may still pose challenges. Future work should integrate multi-faceted error control strategies.
Future Work
Future directions include developing adaptive, data-driven cycle schedules possibly guided by spectral feedback or reinforcement learning. Extending the framework to nonlinear, multi-physics systems and exploring automated parameter tuning will broaden its applicability. Additionally, integrating spectral regularization and uncertainty quantification could further enhance robustness and interpretability, paving the way for real-time, high-fidelity physics simulations in complex scientific and engineering contexts.
AI Executive Summary
Simulating physical systems governed by partial differential equations (PDEs) is fundamental across science and engineering, yet traditional numerical methods often demand immense computational resources. Deep learning surrogates, especially transformer-based models like ViT, have emerged as promising alternatives, offering faster inference once trained. However, these models typically rely on fixed patch sizes, which introduce systematic spectral errors, notably at harmonic frequencies, leading to artifacts and long-term instability in autoregressive predictions. Such limitations hinder their deployment in resource-constrained or real-time scenarios.
To address this, the paper introduces Overtone, a novel framework that leverages cyclic patch size modulation during inference. This approach involves two modules—CSM, which dynamically adjusts convolution stride, and CKM, which resizes kernels via bicubic interpolation—both operating independently of the backbone architecture. During autoregressive rollouts, these modules cyclically alternate patch sizes (e.g., 4, 8, 16), effectively dispersing spectral errors across the frequency spectrum, preventing their accumulation at specific harmonic frequencies. This spectral error distribution reduces artifacts and enhances prediction stability over long horizons.
Empirical evaluations on challenging 2D and 3D PDE benchmarks demonstrate that models employing Overtone achieve up to 40% lower variance-normalized RMSE (VRMSE) compared to fixed-patch models, across various inference compute budgets. Notably, a single flexible model can match or outperform multiple fixed-patch models trained separately at different scales, without retraining. The approach's architecture-agnostic nature ensures broad applicability, including vanilla and axial ViT architectures.
This work significantly advances the field by providing a practical, compute-adaptive solution that improves long-term stability and spectral error management in physics emulation. Its ability to dynamically balance accuracy and computational cost at inference makes it highly relevant for real-world scientific computing, where resource constraints and real-time demands are critical. Future work aims to refine cycle scheduling, extend to multi-physics systems, and incorporate spectral feedback mechanisms, promising a new paradigm for scalable, stable physics-informed AI models.
Deep Dive
Abstract
Transformer-based PDE surrogates achieve remarkable performance but face two key challenges: fixed patch sizes cause systematic error accumulation at harmonic frequencies, and computational costs remain inflexible regardless of problem complexity or available resources. We introduce Overtone, a unified solution through dynamic patch size control at inference. Overtone's key insight is that cyclically modulating patch sizes during autoregressive rollouts distributes errors across the frequency spectrum, mitigating the systematic harmonic artifact accumulation that plague fixed-patch models. We implement this through two architecture-agnostic modules--CSM (using dynamic stride modulation) and CKM (using dynamic kernel resizing)--that together provide both harmonic mitigation and compute-adaptive deployment. This flexible tokenization lets users trade accuracy for speed dynamically based on computational constraints, and the cyclic rollout strategy yields up to 40% lower long rollout error in variance-normalised RMSE (VRMSE) compared to conventional, static-patch surrogates. Across challenging 2D and 3D PDE benchmarks, one Overtone model matches or exceeds fixed-patch baselines across inference compute budgets, when trained under a fixed total training budget setting.