L4GM: Large 4D Gaussian Reconstruction Model

Recent years have seen significant progress in 3D scene reconstruction, driven by neural radiance fields (NeRF), generative models, and large-scale datasets like Objaverse. Static 3D reconstruction from single images has become fast and accurate, exemplified by models like LGM. However, extending these methods to dynamic scenes, especially from monocular videos, remains challenging due to data scarcity, computational costs, and the need for temporal consistency. Existing approaches often rely on multi-view setups or iterative optimization, which are slow and impractical for real-time applications. Synthetic datasets have been used to pretrain models, but generalization to real-world videos is limited. Addressing these gaps, this work aims to develop a fast, scalable, and generalizable method for single-view 4D reconstruction.

The core challenge is to reconstruct temporally consistent, high-fidelity 4D animations from monocular videos in real-time. Existing methods are either too slow, requiring hours of optimization, or lack the ability to produce smooth, detailed motion. The problem is compounded by the limited information in single-view videos, occlusions, and the need for models to generalize across diverse object categories and motion types. Achieving a balance between speed, quality, and generalization remains a key bottleneck in the field.

This paper introduces several key innovations: 1) leveraging a large-scale synthetic animation dataset for training, 2) extending a pretrained 3D Gaussian model (LGM) to process temporal sequences, 3) integrating temporal self-attention layers within a U-Net architecture to model dynamic scenes, 4) employing low-fps sampling combined with learned high-fps interpolation, and 5) designing a fast, feed-forward inference pipeline that produces high-quality 4D reconstructions in seconds. These innovations collectively enable real-time, detailed, and generalizable 4D scene synthesis from monocular videos.

• �� Start with a pretrained LGM model that outputs 3D Gaussian ellipsoids from multiview images.
• �� Generate initial multiview images from the first video frame using a diffusion-based view synthesis model (ImageDream).
• �� Reconstruct an initial 3D Gaussian scene from these views, then replicate views across time with fixed camera assumptions.
• �� Incorporate temporal self-attention layers in the U-Net to learn scene dynamics across frames.
• �� Sample low-fps frames, then train an interpolation model to produce intermediate 3D Gaussian representations, upsampling to higher fps.
• �� Use multiview rendering losses during training to ensure view consistency.
• �� For long videos, adopt an autoregressive approach, processing chunks sequentially.
• �� Fine-tune the entire pipeline on large synthetic datasets, ensuring robustness and generalization.

The model is trained on the Objaverse-4D dataset, with 8FPS sampling, T=8 frames per chunk, using 4 input and 4 supervision views. Evaluation on the Consistent4D benchmark shows superior performance: LPIPS 0.12, CLIP 0.94, FVD 691.87, with a 3-second inference time. Additional tests on real-world videos from Sora and ActivityNet confirm strong generalization. Ablation studies demonstrate the importance of pretraining, temporal attention, and interpolation modules. The model outperforms prior methods by large margins in both qualitative and quantitative metrics, with significant improvements in detail, motion realism, and computational efficiency.

Quantitative metrics reveal that L4GM achieves LPIPS of 0.12, CLIP similarity of 0.94, and FVD of 691.87, at a runtime of 3 seconds, surpassing existing approaches by over 100 times in speed. Qualitative results show sharp, coherent animations with natural motion, even in complex scenarios. The model generalizes well to in-the-wild videos, producing high-fidelity 3D assets that maintain temporal smoothness. The interpolation module further enhances frame rate, enabling smooth 16FPS animations, which are visually more appealing and suitable for real-time applications.

This technology can be directly applied to virtual reality content creation, game asset generation, and film production, enabling rapid conversion of monocular videos into animated 3D assets. It reduces manual effort, accelerates workflows, and democratizes high-quality 3D content creation. Future integration with multi-camera setups and scene understanding could extend its use to live streaming, AR/VR applications, and interactive media, transforming how digital content is produced and consumed.

The model's reliance on synthetic training data may limit performance in highly complex, real-world scenarios with occlusions, lighting variations, or non-rigid deformations. Assumption of static cameras restricts its applicability to scenes with camera motion. Interpolation quality may degrade during rapid movements, and the current architecture may struggle with multi-object interactions or scenes with significant non-rigid transformations. Addressing these limitations requires further research into adaptive modeling and scene understanding.