MVFusion-GS: Motion-Variance Guided Temporal Attention for High-Quality Dynamic Gaussian Splatting
MVFusion-GS integrates motion-variance guided features and Transformer-based temporal attention to enhance dynamic Gaussian scene reconstruction, outperforming SOTA by 1.2dB PSNR.
Key Findings
Methodology
This approach combines a motion-variance guided refinement module with a Transformer-based MotionFormer Temporal Attention. The method samples deformation trajectories of each Gaussian across multiple timestamps, computes position, rotation, and scale statistics to form a 13D global motion signature, serving as a motion prior. The MotionFormer module employs cross-attention over neighboring time steps to model local motion dependencies, enhancing temporal consistency. These mechanisms are integrated into the deformation network, enabling explicit motion awareness for better dynamic-static separation. The training involves staged optimization of geometry, deformation, and motion modules, validated on datasets like NeRF On-the-Go and WildGaussians, with metrics including PSNR, SSIM, and LPIPS.
Key Results
- On the NeRF On-the-Go dataset, the proposed method achieves an average PSNR improvement of 1.2dB over DeGauss, with a 0.02 reduction in LPIPS, demonstrating superior background clarity and dynamic detail capture. Ablation studies show that the global trajectory signature contributes approximately 0.4dB, while the temporal attention adds 0.3dB, with combined effects yielding the best performance. The method effectively reduces pseudo-static residuals, especially in scenes with subtle or transient motions, outperforming existing methods such as WildGaussians and SpotlessSplats.
- Across multiple complex scenes, the approach maintains high temporal consistency, accurately modeling subtle motions and fast movements, with fewer artifacts and clearer dynamic object details. Quantitative metrics confirm the robustness of the model in diverse scenarios, with consistent improvements in both static background reconstruction and dynamic foreground fidelity.
- The experimental results validate that explicit motion priors and local temporal modeling significantly enhance the quality of dynamic scene reconstruction, establishing a new state-of-the-art in real-time, high-fidelity 3D scene synthesis.
Significance
This work addresses a fundamental challenge in dynamic scene reconstruction: capturing subtle and transient motions while maintaining static background clarity. By integrating explicit motion priors with Transformer-based temporal modeling, the method overcomes limitations of previous approaches that relied solely on implicit deformation fields. The resulting system achieves more accurate motion modeling, reduces artifacts, and improves visual fidelity, which is crucial for applications in virtual reality, film production, and autonomous systems. The proposed framework paves the way for future research in scalable, real-time dynamic scene understanding, with potential extensions to multi-modal data integration and adaptive temporal modeling, fostering a new era of high-quality, real-time 3D scene synthesis.
Technical Contribution
The paper introduces a novel combination of motion-variance guided global trajectory signatures with Transformer-based short-term temporal attention within a deformation network for dynamic Gaussian scene reconstruction. The key technical innovations include: 1) a 13D global motion signature capturing long-term motion statistics, injected into the deformation latent space; 2) a lightweight local variance dictionary providing instantaneous motion cues; 3) a MotionFormer Temporal Attention module employing cross-attention over neighboring frames to model local motion dependencies. These components are integrated seamlessly, enabling the network to distinguish subtle motions and transient dynamics effectively. The approach improves upon prior methods like DeGauss by explicitly modeling motion, leading to enhanced dynamic fidelity and static background purity, while maintaining real-time rendering capabilities.
Novelty
This work is the first to explicitly incorporate global motion trajectory signatures and local temporal attention into Gaussian-based deformation networks for dynamic scene reconstruction. Unlike prior approaches that rely solely on implicit deformation fields, this method leverages explicit motion priors and short-term temporal dependencies, resulting in more accurate motion modeling, especially for subtle and transient movements. The integration of a 13D global signature with Transformer-based attention represents a significant step forward, providing a comprehensive and scalable framework for high-fidelity, real-time dynamic scene synthesis. This dual-mechanism approach sets a new benchmark in the field, bridging the gap between global motion understanding and local temporal coherence.
Limitations
- The model's performance may degrade in scenes with extremely rapid or complex deformations beyond the current sampling and modeling capacity, due to limitations in trajectory estimation and fixed temporal window size.
- Computational overhead increases with scene complexity, impacting real-time applicability in highly dynamic environments.
- The reliance on sampled deformation trajectories and fixed window sizes may limit generalization to unseen motion patterns or highly non-linear dynamics, requiring further adaptive mechanisms.
Future Work
Future research will focus on adaptive temporal windowing and multi-scale motion modeling to better capture complex, non-linear dynamics. Incorporating learned trajectory prediction models could improve robustness against rapid movements. Additionally, integrating multi-modal cues such as optical flow and depth information may further enhance motion understanding. The ultimate goal is to develop a fully end-to-end, real-time dynamic scene reconstruction system capable of handling diverse and complex motions with minimal artifacts, enabling applications in immersive VR, autonomous navigation, and real-time content creation.
AI Executive Summary
The rapid development of virtual and augmented reality technologies has heightened demand for high-fidelity dynamic scene reconstruction. Traditional methods, primarily designed for static scenes, struggle to accurately model motion, often resulting in artifacts like residual pseudo-static residuals and blurred dynamic details. Recent advances such as 3D Gaussian Splatting (3DGS) have enabled real-time rendering, but their extension to dynamic scenes remains challenging due to insufficient motion awareness.
This paper introduces MVFusion-GS, a novel framework that significantly advances dynamic scene reconstruction by explicitly modeling motion. The core innovation lies in integrating a motion-variance guided refinement module with a Transformer-based MotionFormer Temporal Attention. The former samples deformation trajectories across time, computes statistical features—such as position, rotation, and scale variances—and encodes them into a 13-dimensional global motion signature. This signature serves as a prior, guiding the deformation network to better distinguish between static and dynamic elements. The latter module employs cross-attention over neighboring frames, capturing local temporal dependencies to improve deformation consistency.
Experimental results on datasets like NeRF On-the-Go and WildGaussians demonstrate that MVFusion-GS outperforms existing methods such as DeGauss and SpaceTimeGS. Quantitative metrics show an average PSNR improvement of approximately 1.2dB, with notable reductions in perceptual errors (LPIPS). Visual comparisons reveal clearer backgrounds, more detailed dynamic objects, and fewer artifacts, especially in scenes with subtle or transient motions. These improvements highlight the importance of explicit motion modeling in high-quality 3D scene synthesis.
The significance of this work extends beyond technical novelty. It addresses a fundamental bottleneck in real-time dynamic scene reconstruction, enabling applications in immersive VR, film production, and autonomous systems. By explicitly capturing motion cues, the approach enhances both the fidelity and stability of reconstructed scenes, paving the way for more realistic virtual environments.
Despite its successes, the method faces limitations such as increased computational costs and challenges in scenes with extremely rapid or complex motions. Future work aims to incorporate adaptive temporal modeling, multi-modal cues, and more robust trajectory estimation to further improve performance. Overall, MVFusion-GS marks a substantial step toward real-time, high-fidelity dynamic scene understanding, with broad implications for both academia and industry.
Deep Dive
Abstract
3D Gaussian Splatting (3DGS) enables real-time novel view synthesis for static scenes. Extending it to dynamic scenes via deformation fields has recently attracted significant attention, particularly for dynamic scene reconstructionband distractor-free. However, existing deformation networks lack explicit motion awareness: they neither capture long-term motion intensity nor exploit short-term temporal coherence, leading to inaccurate foreground deformation and pseudo-static residuals in the background. We present MVFusion-GS, a method that enhances deformation networks with two complementary motion-aware mechanisms. The Motion-Variance Guided Refinement aggregates per-Gaussian deformation statistics across time to estimate motion variance and uses it to guide dynamic-static separation during deformation prediction. The MotionFormer Temporal Attention module applies Transformer self-attention over neighboring timesteps to model local motion dependencies and improve temporal consistency. Extensive experiments on both dynamic scene reconstruction and distractor-free reconstruction benchmarks demonstrate state-of-the-art performance, showing that explicit motion awareness improves both foreground motion modeling and static background reconstruction.