Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
Dynamic Gaussian model enables real-time 6-DOF scene tracking and novel-view synthesis without correspondence inputs, achieving 28.7 PSNR and 850 FPS.
Key Findings
Methodology
This paper extends static 3D Gaussian splatting into a dynamic framework by allowing Gaussian centers and orientations to vary over time, regularized with local rigidity and rotational similarity constraints. Using a differentiable renderer, the model optimizes scene parameters directly from multi-view videos without requiring explicit correspondence or optical flow. The approach employs a set of physical priors to ensure spatial consistency, with parameters including position, quaternion rotation, size, color, and opacity. The system is trained on multi-camera datasets like CMU Panoptic, achieving high-quality dynamic scene reconstruction and dense 6-DOF tracking through end-to-end gradient descent, with real-time rendering at 850 FPS.
Key Results
- On 27-camera multi-view datasets, the model achieves 28.7 PSNR in dynamic novel view synthesis, with a tracking error of only 2.21cm over 150 frames, and renders at 850 FPS. It surpasses previous methods by a factor of 10 in accuracy, demonstrating robust performance in complex non-rigid scenes. The model accurately captures scene rotations and deformations, enabling applications like scene editing and 4D video generation.
- Experiments validate that the physical regularizations (rigidity, rotation, isometry) significantly improve long-term consistency and physical plausibility. The model maintains high fidelity across diverse motions, occlusions, and scene complexities, outperforming baselines in both visual quality and tracking precision.
- The approach supports full 6-DOF dense tracking, including rotation of individual points, and enables scene manipulations such as object insertion and camera following. Its efficiency and accuracy open new avenues for real-time AR/VR, film VFX, and robotics, with potential for large-scale scene understanding.
Significance
This work advances the state-of-the-art in dynamic scene modeling by eliminating the need for explicit correspondence or flow inputs, relying solely on differentiable rendering. It offers a scalable, fast, and physically plausible solution for dense 6-DOF tracking and novel view synthesis, crucial for immersive AR/VR, intelligent robotics, and content creation. The ability to perform real-time, high-fidelity reconstruction of complex scenes addresses longstanding challenges in dynamic scene understanding, paving the way for more autonomous and interactive virtual environments.
Technical Contribution
The core innovation is the integration of oriented Gaussian particles with physical regularization, enabling full 6-DOF scene tracking. The use of a differentiable Gaussian splatting renderer allows direct optimization of scene parameters from images, bypassing traditional correspondence or flow reliance. The model’s physical priors enforce local rigidity and long-term consistency, ensuring physically plausible motion. This approach significantly improves robustness, speed, and accuracy over previous neural scene representations, and supports real-time rendering at 850 FPS.
Novelty
This is the first work to extend Gaussian splatting to dynamic scenes with full 6-DOF dense tracking, combining physical regularization with end-to-end differentiable optimization. Unlike prior methods relying on optical flow or static canonical models, this approach models scene elements as oriented particles with persistent attributes, enabling accurate, real-time reconstruction without explicit correspondence inputs. It introduces a new paradigm for scalable, physically grounded dynamic scene modeling.
Limitations
- The method may struggle with scenes involving extreme occlusion or rapid, unpredictable motions, where regularization may not suffice to maintain tracking accuracy.
- Computational costs increase with scene complexity and Gaussian density, potentially limiting scalability to very large environments.
- Currently optimized for multi-view static camera setups; adapting to moving cameras or monocular inputs remains an open challenge.
Future Work
Future directions include integrating learned physical priors for more complex scene dynamics, extending to monocular or moving camera scenarios, and improving scalability for larger scenes. Combining this approach with semantic understanding could enable scene-aware editing and interaction, further broadening its application scope in AR/VR, robotics, and content creation.
AI Executive Summary
This paper introduces a groundbreaking dynamic scene modeling framework based on an extended 3D Gaussian splatting approach. Unlike previous methods that depend heavily on explicit correspondence, optical flow, or canonical models, this method leverages a set of oriented Gaussian particles whose positions and orientations evolve over time, regularized by physically motivated constraints such as local rigidity and rotational similarity. The core idea is to optimize scene parameters directly from multi-view videos via differentiable rendering, enabling high-fidelity, real-time reconstruction and dense 6-DOF tracking without any prior correspondence inputs.
The system employs a physically grounded regularization strategy, ensuring that scene elements move in a plausible manner consistent with rigid or near-rigid transformations. This approach allows the model to maintain spatial and rotational coherence over long sequences, even under complex non-rigid motions. The experimental results on the CMU Panoptic dataset demonstrate remarkable performance: achieving 28.7 PSNR in novel view synthesis, with a tracking error of only 2.21cm over 150 frames, and rendering at 850 FPS. These results significantly outperform prior state-of-the-art methods, which often rely on optical flow or require extensive training.
The implications of this work are profound. It provides a scalable, fast, and physically plausible solution for dynamic scene reconstruction, opening new possibilities in AR/VR, film VFX, and robotics. The ability to perform real-time, high-precision 6-DOF tracking and scene editing from multi-view videos addresses longstanding challenges in dynamic scene understanding. Moving forward, integrating learned physical priors, handling monocular or moving camera setups, and scaling to larger scenes are promising directions. Overall, this work marks a major step toward autonomous, interactive virtual environments with high fidelity and physical realism.
Deep Dive
Abstract
We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians which are optimized to reconstruct input images via differentiable rendering. To model dynamic scenes, we allow Gaussians to move and rotate over time while enforcing that they have persistent color, opacity, and size. By regularizing Gaussians' motion and rotation with local-rigidity constraints, we show that our Dynamic 3D Gaussians correctly model the same area of physical space over time, including the rotation of that space. Dense 6-DOF tracking and dynamic reconstruction emerges naturally from persistent dynamic view synthesis, without requiring any correspondence or flow as input. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.