Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction

Reconstructing dynamic scenes in four dimensions from monocular videos has long been a fundamental challenge in computer vision. Traditional approaches relied heavily on scene-specific optimization, which, while capable of high-fidelity results, suffered from high computational costs, limited scalability, and poor generalization to new scenes. These methods often required hours of optimization per scene, making them impractical for large-scale or real-time applications. Recent advances in neural rendering, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), introduced powerful representations capable of high-quality static scene reconstruction. Extending these to dynamic scenes, however, posed additional challenges: how to model scene motion coherently across time, how to handle occlusions and view-dependent biases, and how to do so efficiently without scene-specific tuning.

In this context, the paper introduces C4G, a novel framework that leverages a set of timestamp-conditioned learnable Gaussian query tokens. These tokens are decoded via a transformer decoder with full self-attention, enabling the model to aggregate global multi-frame features into a compact set of 3D Gaussians representing the scene at any arbitrary time. This design effectively addresses the redundancy and view-dependence issues prevalent in pixel-wise high Gaussian count methods. The key innovation is conditioning the Gaussian queries on timestamps, which allows the model to produce temporally coherent scene representations without scene-specific optimization.

The authors further enhance the visual quality of the reconstructed scenes by integrating a video diffusion model-based rendering refinement module. This module refines the rendered images conditioned on input views, filling in high-frequency details and reducing artifacts. Moreover, the model employs a feature lifting mechanism, mapping 2D foundation model features into a 4D feature field, supporting point tracking and scene understanding tasks. The entire system is trained end-to-end, relying solely on photometric and auxiliary supervision signals like depth, normals, and motion tracking, without requiring camera pose information.

Experimental results demonstrate that C4G achieves state-of-the-art or competitive performance across multiple dynamic scene datasets, including DynaCheck, TUM-Dynamics, and NVIDIA. It significantly outperforms existing methods in novel view synthesis, maintaining high PSNR scores (~15.64dB) with far fewer Gaussians (thousands versus hundreds of thousands). The model exhibits robust temporal interpolation capabilities, with PSNR only slightly decreasing even at large temporal gaps (∆t=8). Point tracking and 4D feature field evaluations confirm that C4G captures scene-wide motion trajectories more accurately than pixel-wise approaches, validating its understanding of global scene dynamics.

This work represents a major step forward in dynamic scene reconstruction, offering a scalable, generalizable, and efficient solution. Its ability to model scenes with fewer primitives, without scene-specific optimization, opens new possibilities for real-time applications in AR/VR, robotics, and content creation. The integration of diffusion-based refinement bridges the gap between geometric accuracy and visual fidelity, setting a new standard for neural dynamic scene modeling. Nonetheless, challenges remain in handling extremely fast motions, occlusions, and large-scale scenes, which motivate future research directions such as multi-scale attention, multi-modal cues, and unsupervised learning strategies.