Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models
This paper introduces Riemannian flow matching in VGGT's latent space for 3D scene generation, outperforming baselines with improved geometric consistency.
Key Findings
Methodology
The authors define VGGT's latent space as a product of four zero-mean hyperspheres and propose a conditional Riemannian flow matching (RFM) model. The model learns a velocity field on this manifold, conditioned on sparse unposed views and target camera pose, to generate consistent 3D scene representations. The approach leverages geometric mappings (exponential and logarithm maps) to ensure latent vectors remain on the manifold, avoiding mode collapse common in Euclidean flow models. The architecture integrates a multi-scale transformer with attention mechanisms, conditioning on view and pose information, and employs a Riemannian ODE solver for sampling. Decoders reconstruct depth, point clouds, and RGB images from generated latents, enabling high-quality scene synthesis.
Key Results
- On datasets RealEstate10K, ScanNet++, and ETH3D, the proposed method achieved state-of-the-art results, reducing depth RMSE to 0.191 (ScanNet++ high overlap) and increasing RGB PSNR to 30.72dB. FID scores improved significantly, demonstrating better visual quality and geometric accuracy compared to DepthSplat and Gen3R.
- In sparse-view scenarios (1-2 views), the model maintained high geometric fidelity, outperforming baselines in both depth and RGB metrics, especially under low overlap conditions.
- Ablation studies confirmed that Riemannian flow matching prevents mode collapse, ensuring stable, diverse scene generation across different input configurations.
Significance
This work advances the application of geometric foundation models by integrating Riemannian geometry into latent space flow modeling, enabling probabilistic, multi-view consistent 3D scene generation. It addresses the longstanding challenge of generating plausible 3D scenes from sparse, uncalibrated views, which is critical for robotics, AR/VR, and autonomous driving. The approach leverages pre-trained geometric priors, reducing the need for extensive scene-specific training, and opens new avenues for scalable, high-fidelity scene synthesis.
Technical Contribution
The key technical innovation is the formulation of a conditional flow matching algorithm on a product manifold of hyperspheres, respecting the intrinsic geometry of VGGT's latent space. The model employs exponential and logarithm maps for geodesic interpolation, combined with a multi-scale transformer conditioned on view and pose features. This enables stable, geometry-aware probabilistic modeling, avoiding mode collapse typical of Euclidean approaches. The architecture supports end-to-end training and inference, producing high-quality depth, point clouds, and RGB images from sparse inputs.
Novelty
This is the first work to embed flow matching directly within the hyperspherical latent space of a geometric foundation model, exploiting its manifold structure for probabilistic scene generation. Unlike prior methods that treat latent vectors as Euclidean, this approach respects the intrinsic geometry, leading to more stable and diverse outputs. It bridges the gap between geometric priors and generative modeling, offering a new paradigm for 3D scene synthesis.
Limitations
- The approach relies heavily on pre-trained VGGT models, limiting generalization to scenes outside the training distribution. Computational costs for sampling (about 0.88 seconds per sample) hinder real-time applications.
- Current framework handles static scenes; dynamic scene modeling remains an open challenge. The method's performance in highly complex or occluded environments needs further validation.
- The model's scalability to larger scenes or real-time scenarios requires optimization of the Riemannian ODE solver and network architecture.
Future Work
Future directions include extending the framework to dynamic scenes with temporal modeling, improving sampling efficiency for real-time deployment, and integrating multi-modal data such as LiDAR or audio. Exploring more efficient Riemannian solvers and scalable architectures will be key to broader adoption. Additionally, investigating unsupervised or semi-supervised training strategies could enhance generalization to diverse environments.
AI Executive Summary
The demand for accurate 3D scene understanding from sparse, uncalibrated images remains a significant challenge in computer vision, impacting applications from robotics to AR/VR. Existing methods like NeRF and Gaussian Splatting excel in scene reconstruction but require extensive per-scene optimization, limiting scalability. Geometric foundation models such as VGGT have introduced a paradigm shift by providing robust 3D priors through large-scale pretraining, enabling direct inference of scene geometry and camera parameters. However, their deterministic nature constrains their ability to generate diverse, plausible scenes beyond observed views.
This paper proposes a novel approach that bridges the gap between deterministic geometric priors and probabilistic scene generation. The core innovation lies in performing flow matching directly in VGGT’s latent space, which is modeled as a product of four hyperspheres. Recognizing the manifold structure of these latent tokens, the authors develop a conditional Riemannian flow matching (RFM) framework. This approach respects the intrinsic geometry, employing geodesic-based exponential and logarithm maps to interpolate and transport latent vectors without collapsing modes. The model uses a multi-scale transformer conditioned on sparse observations and target pose, guiding the generation process within the geometric constraints.
Experimental results on datasets like RealEstate10K, ScanNet++, and ETH3D demonstrate that this method significantly outperforms existing baselines such as DepthSplat and Gen3R in depth accuracy and visual quality. The approach maintains high geometric fidelity even with limited input views, validating the effectiveness of the Riemannian geometry-aware probabilistic modeling. The generated scenes exhibit coherent geometry and appearance, confirming the potential of latent space flow matching on geometric foundation models.
Overall, this work introduces a powerful new paradigm for 3D scene generation, leveraging the geometric structure of learned latent spaces. It opens avenues for scalable, high-fidelity scene synthesis in various real-world applications, from autonomous navigation to immersive virtual environments. Despite current computational costs and static scene limitations, the framework paves the way for future innovations in probabilistic 3D modeling grounded in geometric deep learning.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭，但只拿到几块食材（图片），你想做出一盘完整的菜（3D场景）。传统的方法就像是逐一尝试每个食材的搭配，费时费力，还不一定能做出好菜。而这项新技术像是有个聪明的厨师，他提前知道各种食材的搭配规则（几何先验），只用少量原料，就能快速拼出一盘色香味俱佳的菜（完整场景）。他用一种特别的“魔法”把原料变成完整的菜肴，不仅快，还很有创意。它就像用几块拼图拼出一幅完整的画（场景），而且画得非常逼真、合理。这样，即使只看到少量碎片，也能拼出一幅完整的场景，就像魔法一样！
ELI14 Explained like you're 14
想象你在玩拼图游戏，但只剩下一两块碎片。你想知道完整的图片是什么样子。以前的方法就像拼图时试图把每一块都放到正确位置，但有时候会卡住或者拼错。现在，这个新方法像是有个聪明的朋友，他知道图片的整体轮廓和颜色，能帮你用几块碎片拼出一幅完整的画。他用一种特别的数学技巧（黎曼流匹配），确保每一块都放得既漂亮又合理。这个朋友还会根据你给的几块碎片，预测出其他缺失的部分，让你看到一幅完整的画。这样，即使只看到少量碎片，也能拼出一幅逼真的场景，就像魔法一样！
Abstract
Geometric foundation models, such as the Visual Geometry Grounded Transformer (VGGT), provide strong 3D priors from unposed images. However, such models operate purely in a feed-forward, deterministic regime, \ie~they cannot generate plausible geometry beyond what the input views directly support. Generative models for 3D scenes, on the other hand, must rely on strong geometric priors to produce coherent outputs from sparse inputs. We bridge these two paradigms by performing flow matching directly in VGGT's latent space, leveraging its learned 3D priors without committing to any explicit downstream representation such as Gaussians, meshes, or video-VAE latents. This requires respecting the latent geometry: VGGT tokens occupy a product of high-dimensional hyperspheres on which standard Euclidean flow matching fails. We address this with a Riemannian Flow Matching framework defined on a product manifold of four hyperspheres, aligned with VGGT's multi-scale encoder, which keeps generated tokens on the valid data manifold required by the frozen decoding heads. On RealEstate10K, ScanNet++ and ETH3D, our method achieves strong performance against recent scene generation baselines in both per-view appearance and aggregated 3D geometry, establishing latent-space flow matching on geometric foundation models as a viable paradigm for 3D generation. The project page can be found $\href{https://lisaweijler.github.io/geometry-grounded-rfm/}{\text{here}}$.