SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations

Novel view synthesis technology plays a crucial role in computer vision and graphics, supporting immersive experiences in fields like video games and mixed reality. Although neural reconstruction techniques have made significant progress recently, they typically rely on dense multi-view observation data, facing numerous limitations in practical applications. Therefore, this paper focuses on how to achieve high-quality 3D scene reconstruction and synthesize realistic novel views from sparse or even single-view observation.

The core problem in sparse view reconstruction is accurately reconstructing complex and diverse real-world scenes with limited observation data. Sparse captures leave occluded regions unseen, and geometric cues are too few to constrain reconstruction. Existing methods struggle to generalize to complex scenarios, and reconstruction failures under large viewpoint changes can severely affect novel view synthesis quality.

SpatialCrafter's core innovation lies in leveraging the rich physical world knowledge embedded in video diffusion models to provide plausible additional observations for scene reconstruction, effectively reducing problem complexity. By introducing camera parameterization and epipolar attention mechanisms, it enhances camera control and 3D consistency in generated videos. Additionally, a unified scale estimation strategy resolves scale ambiguity in multi-dataset training.

• �� Use video diffusion models to generate additional views, alleviating ambiguity in sparse view reconstruction.
• �� Achieve precise camera control through a trainable camera encoder.
• �� Introduce epipolar attention mechanisms to enhance 3D consistency between video frames.
• �� Integrate monocular depth priors with semantic features in the video latent space to directly regress 3D Gaussian primitives.
• �� Efficiently process long-sequence features using a hybrid network structure.

Experiments were conducted on datasets such as RealEstate10K, ACID, and DL3DV-10K to evaluate the method's generalization across different scenarios. Metrics like FVD, FID, PSNR, and SSIM were used to assess visual quality and camera control precision. Ablation studies confirmed the contributions of camera parameterization and epipolar attention modules to enhancing 3D consistency.

Experimental results show that SpatialCrafter performs excellently in sparse view reconstruction. On the RealEstate10K dataset, the FVD metric decreased from 22.65 to 18.25, indicating significant visual quality improvement. On the Tanks-and-Temples dataset, PSNR improved from 14.64 to 16.12, and SSIM increased from 0.388 to 0.501.

The method can be applied in scenarios requiring 3D scene reconstruction from sparse views, such as virtual reality, augmented reality, and film production. By generating additional views, it enhances reconstruction accuracy and consistency, expanding applicability in real-world scenarios.

While the method performs excellently in sparse view reconstruction, generated video frames may contain low-quality areas in large-scale outdoor environments, affecting reconstruction stability. Additionally, recovering poses and sparse point clouds in stylized scenes is challenging. Future work could explore improving reconstruction accuracy in more complex scenes and enhancing model real-time performance without increasing computational costs.