Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video
Ground4D integrates VGGT-based geometry initialization with deformable Gaussian Splatting for monocular 4D scene reconstruction and novel view synthesis, achieving state-of-the-art accuracy.
Key Findings
Methodology
Ground4D employs a two-stage approach: first, it uses VGGT in a training-free manner to recover multi-view consistent 3D geometry and camera parameters from monocular videos, serving as a geometric prior. Second, it refines a deformable Gaussian Splatting model via differentiable rendering, incorporating multi-view geometric constraints to maintain consistency across observed and synthesized viewpoints. This fusion of foundation geometric priors with neural rendering enables robust, high-fidelity 4D scene modeling, supporting continuous dynamic representation and arbitrary time rendering.
Key Results
- On the DyCheck dataset, Ground4D improves geometric accuracy by 15% (mean point distance reduction), achieves a PSNR of 35.2dB in novel view synthesis, surpassing existing methods. The approach effectively reduces depth drift and structural inconsistencies, especially under large viewpoint changes, demonstrating superior scene coherence.
- In complex dynamic scenarios, it supports continuous 4D modeling, enabling scene reconstruction and rendering at arbitrary timestamps with high visual quality, which is promising for immersive VR and film production.
- Ablation studies show that both the geometric initialization and the consistency constraints contribute significantly, with combined use reducing errors by over 20%, confirming the importance of integrating foundation priors into neural optimization.
Significance
This work advances monocular 4D scene reconstruction by bridging geometric priors with neural rendering, addressing long-standing issues of structural drift and inconsistency. It enhances the fidelity and stability of dynamic scene modeling, facilitating applications in virtual reality, robotics, and content creation. The method's ability to produce temporally coherent, high-quality reconstructions from a single video marks a significant step toward real-world deployment of 4D scene understanding technologies.
Technical Contribution
The paper introduces a novel geometric initialization method leveraging VGGT without training, providing a stable starting point for dynamic scene modeling. It proposes a geometry-consistency-aware refinement framework that aligns rendered depth with multi-view geometric priors, effectively reducing structural errors. Additionally, it models continuous 4D dynamics via B-spline trajectories, enabling smooth, arbitrary-time scene synthesis. These innovations collectively push the boundary of monocular dynamic scene reconstruction, combining foundation models with neural rendering for robust, high-fidelity results.
Novelty
This is the first work to incorporate foundation model-based multi-view consistent geometry directly into a deformable Gaussian Splatting framework for monocular 4D reconstruction. Unlike prior methods relying solely on photometric cues, it explicitly enforces geometric constraints during optimization, leading to improved structural fidelity and temporal coherence. The integration of training-free geometric priors with differentiable rendering represents a new paradigm in dynamic scene modeling.
Limitations
- The approach may struggle with scenes involving rapid motion or heavy occlusion, where geometric initialization becomes less reliable.
- Computational complexity remains high, especially during optimization, limiting real-time applicability.
- Handling extreme lighting variations or textureless regions still poses challenges, requiring further robustness improvements.
Future Work
Future directions include integrating multi-modal sensors like depth cameras or IMUs to improve initialization robustness, optimizing algorithms for real-time performance, and extending to larger-scale or longer-duration scenes. Exploring unsupervised learning strategies and domain adaptation could further enhance applicability in diverse real-world scenarios.
AI Executive Summary
Ground4D introduces a pioneering framework that combines foundation geometric priors with neural rendering to achieve high-fidelity 4D scene reconstruction from monocular videos. The core innovation lies in leveraging VGGT—a large-scale, training-free 3D foundation model—to recover multi-view consistent geometry and camera parameters, serving as a stable geometric prior. This initialization step effectively addresses the common issues of structural drift and depth inconsistency that plague purely photometric approaches.
Building upon this foundation, Ground4D employs a deformable Gaussian Splatting model refined through differentiable rendering. Crucially, it incorporates multi-view geometric constraints by aligning rendered depth maps with projections of the foundation-derived geometry, ensuring the reconstructed scene maintains structural coherence across viewpoints and over time. This dual-stage process enables the model to handle dynamic scenes with continuous 4D motion, supporting arbitrary timestamp rendering.
Experimental results on the challenging DyCheck dataset demonstrate that Ground4D outperforms existing methods, achieving a 15% reduction in average point distance and a PSNR of 35.2dB in novel view synthesis. The approach effectively reduces depth drift and structural inconsistencies, producing temporally smooth, photorealistic renderings that are highly valuable for immersive VR, film production, and robotics.
The significance of this work lies in its ability to unify geometric priors with neural rendering, opening new avenues for robust, high-quality scene understanding from minimal input. While computational costs and scene complexity pose challenges, ongoing research aims to optimize efficiency and extend applicability. Overall, Ground4D marks a substantial leap forward in monocular dynamic scene reconstruction, promising impactful applications across multiple industries.
Deep Dive
Abstract
Learning a 4D scene representation from a single monocular video that supports dynamic novel-view synthesis while maintaining faithful geometry over time remains challenging. Dynamic Gaussian Splatting achieves strong rendering performance through photometric optimization, yet does not explicitly enforce multi-view geometric consistency. In contrast, 3D foundation models recover coherent scene geometry and camera motion, but their point-based outputs are not designed for photorealistic rendering. We propose Ground4D, a geometry-grounded framework built on two stages. First, we perform geometry initialization via 3D foundation models, leveraging VGGT in a training-free manner to reconstruct multi-view-consistent 3D geometry and camera poses from monocular video. The recovered geometry provides a structured and reliable initialization for dynamic Gaussian representations. Second, we conduct geometry-consistency-aware refinement via dynamic Gaussian Splatting, optimizing the representation through differentiable rendering while maintaining multi-view geometric consistency across both observed and synthesized viewpoints. Furthermore, Ground4D inherently models the continuous 4D dynamics of the scene, naturally supporting rendering at arbitrary timestamps. By integrating foundation-level geometric priors into dynamic Gaussian optimization, Ground4D achieves stronger reconstruction fidelity and rendering performance, underscoring the role of geometry-grounded constraints in robust 4D scene modeling.