Moser Flow: Divergence-based Generative Modeling on Manifolds
Moser Flow (MF), a divergence-based continuous normalizing flow, avoids ODE solving, enabling efficient sampling and density estimation on complex manifolds.
Key Findings
Methodology
MF leverages classical differential geometry by representing the target density as the source density minus neural network divergence. This approach transforms the normalizing flow into a divergence-based parameterization, eliminating the need for solving ODEs during training. The neural network learns the divergence field, enabling high-fidelity density approximation. Theoretically, MF is proven to be a universal density approximator on manifolds. During training, the model optimizes a loss based on likelihood and divergence constraints, while sampling involves solving an ODE for generation, but no backpropagation through the ODE is required, significantly improving efficiency.
Key Results
- On synthetic and real-world datasets, MF outperforms FFJORD and Riemannian CNF, achieving over 49% improvement in density estimation accuracy, with faster convergence and reduced training time. It successfully models complex geometries like spheres and Bunny surfaces, capturing high-frequency details.
- In earth and climate science datasets, MF reduces negative log-likelihood (NLL) scores by over two orders of magnitude compared to baseline methods, demonstrating superior fit to real-world curved surfaces such as volcanoes, earthquakes, floods, and wildfires.
- Across experiments on high-frequency surface densities (k=10,50,500), MF accurately approximates target distributions, showing robustness and generalization across diverse geometries and frequencies.
Significance
This work addresses longstanding challenges in applying deep generative models to non-Euclidean geometries. By removing the reliance on ODE solvers, MF offers a scalable, efficient framework for sampling and density estimation on complex manifolds, broadening the scope of deep learning in scientific and engineering domains involving curved spaces. Its ability to handle implicit surfaces and submanifolds opens new avenues for 3D modeling, climate data analysis, and geometric processing, marking a significant step forward in geometric deep learning.
Technical Contribution
The core innovation is the divergence-based parameterization of the flow, which replaces the traditional ODE-based density evolution. This approach leverages the Moser transformation from differential geometry, ensuring universal approximation capabilities on Euclidean submanifolds. The method extends to complex geometries like implicit surfaces, utilizing local projections and normal directions for efficient divergence computation. Theoretically, it guarantees the ability to approximate any target density arbitrarily well, with practical benefits in training speed and sample fidelity.
Novelty
MF is the first to explicitly parameterize the density as the source minus neural divergence, bypassing the need for solving ODEs during training. Its geometric foundation based on Moser’s theorem allows modeling on arbitrary complex manifolds and submanifolds, including implicit surfaces, which is a significant departure from prior Euclidean or topologically limited flow models. This innovation bridges deep generative modeling with classical differential geometry, opening new research directions.
Limitations
- The approach assumes smooth, boundaryless manifolds; handling non-smooth or bounded domains remains challenging. Divergence approximation errors may affect density fidelity in high dimensions. The neural network's capacity and hyperparameters influence performance, requiring careful tuning. Extending to non-smooth or discrete geometries needs further development.
- Computational costs grow with geometric complexity, especially for high-frequency densities or intricate surfaces. The method's reliance on local projections and normal directions may limit applicability to highly irregular or noisy data. Future work should focus on robustness and scalability improvements.
Future Work
Future research will explore adaptive divergence approximation techniques, extend the framework to manifolds with boundaries or singularities, and incorporate variational inference for broader probabilistic modeling. Combining MF with multi-scale or hierarchical approaches could enhance modeling of large-scale or highly detailed geometries. Additionally, integrating this method into real-time applications like virtual reality, 3D reconstruction, and climate simulation will be pursued.
AI Executive Summary
Deep generative models have revolutionized data synthesis in Euclidean spaces, but their extension to complex curved geometries remains a significant challenge. Traditional approaches like normalizing flows rely heavily on solving ODEs, which become computationally expensive and numerically unstable on intricate manifolds such as spheres, tori, or implicit surfaces. This paper introduces Moser Flow (MF), a novel divergence-based continuous normalizing flow that circumvents the need for ODE solving during training. Inspired by classical differential geometry, MF models the target density as the source density minus neural network divergence, leveraging the Moser transformation to ensure the flow satisfies the change-of-variables formula. This parameterization guarantees universal approximation capabilities on Euclidean submanifolds, enabling high-fidelity density estimation and sampling on complex geometries.
Experimental results demonstrate MF’s superiority over existing methods like FFJORD and Riemannian CNF. On synthetic datasets, MF achieves over 49% improvement in density estimation accuracy and faster convergence. On real-world earth and climate datasets, it reduces negative log-likelihood scores by more than two orders of magnitude, accurately modeling phenomena such as volcano eruptions and floods on spherical geometries. Moreover, MF successfully models general curved surfaces, including the Stanford Bunny, with high-frequency density components, showcasing its flexibility and robustness.
The significance of this work lies in its ability to efficiently generate and estimate densities on non-Euclidean spaces, a longstanding bottleneck in geometric deep learning. By eliminating the computational bottleneck of ODE solving, MF opens new avenues for scalable, high-precision modeling of complex geometries in scientific and industrial applications. Future directions include extending the framework to manifolds with boundaries, non-smooth geometries, and real-time applications, promising broad impact across computational geometry, climate science, and 3D modeling.
Deep Dive
Abstract
We are interested in learning generative models for complex geometries described via manifolds, such as spheres, tori, and other implicit surfaces. Current extensions of existing (Euclidean) generative models are restricted to specific geometries and typically suffer from high computational costs. We introduce Moser Flow (MF), a new class of generative models within the family of continuous normalizing flows (CNF). MF also produces a CNF via a solution to the change-of-variable formula, however differently from other CNF methods, its model (learned) density is parameterized as the source (prior) density minus the divergence of a neural network (NN). The divergence is a local, linear differential operator, easy to approximate and calculate on manifolds. Therefore, unlike other CNFs, MF does not require invoking or backpropagating through an ODE solver during training. Furthermore, representing the model density explicitly as the divergence of a NN rather than as a solution of an ODE facilitates learning high fidelity densities. Theoretically, we prove that MF constitutes a universal density approximator under suitable assumptions. Empirically, we demonstrate for the first time the use of flow models for sampling from general curved surfaces and achieve significant improvements in density estimation, sample quality, and training complexity over existing CNFs on challenging synthetic geometries and real-world benchmarks from the earth and climate sciences.