Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos
Proposes Pensieve's two-stage training for high-quality novel view synthesis without calibration, achieving PSNR 26.53 on RealEstate10K.
Key Findings
Methodology
The approach employs a two-stage training framework: first, an implicit scene reconstruction via predicting per-frame latent camera and scene features using a view synthesis model inspired by LVSM, reducing optimization complexity; second, explicit prediction of 3D Gaussian primitives combined with Gaussian splatting rendering loss and depth projection loss to enforce 3D consistency. The model operates solely on raw videos or multi-view images, without requiring camera parameters or priors. The first stage provides a robust initialization by learning scene representations in a latent space, while the second stage aligns these representations with physical 3D geometry, ensuring accurate scene structure and camera pose estimation.
Key Results
- On RealEstate10K, our method achieves PSNR of 26.53, SSIM of 0.843, and LPIPS of 0.115, surpassing baseline methods such as PixelSplat and SelfSplat. Camera pose estimation accuracy also exceeds methods relying on calibration data. The two-stage training accelerates convergence and improves scene consistency. Ablation studies confirm that the explicit geometric alignment in stage two significantly boosts reconstruction fidelity. Interpolated frame strategies further enhance performance under limited input views, especially with only two input frames, reducing artifacts and holes.
- In DL3DV-140, our model attains PSNR of 21.77, outperforming methods like PF3plat and SelfSplat, demonstrating robustness across diverse scenes. The results validate the effectiveness of self-supervised training solely on uncalibrated videos, with high-quality view synthesis and precise camera pose estimation. Quantitative metrics show consistent improvements across multiple benchmarks, confirming the method's generalization and practical viability.
- Additional experiments reveal that the combination of implicit pretraining and explicit geometric alignment yields superior scene fidelity and pose accuracy, especially in challenging scenarios with sparse views or dynamic scenes. The interpolation-based inference strategy notably reduces holes and artifacts, further stabilizing the output quality.
Significance
This work addresses a fundamental bottleneck in 3D scene understanding: reliance on calibrated cameras and geometric priors. By enabling self-supervised learning from uncalibrated videos, it broadens the applicability of scene reconstruction and novel view synthesis to large-scale, real-world datasets. The proposed framework reduces data preparation costs, simplifies deployment, and opens new avenues for applications in virtual reality, film production, and autonomous navigation. Its ability to jointly estimate camera poses and reconstruct scenes without supervision marks a significant advance in unsupervised 3D vision, bridging the gap between research and practical deployment.
Technical Contribution
The core technical innovation lies in the integration of a two-stage training process: stage one employs a LVSM-inspired implicit scene reconstruction that predicts per-frame latent features, reducing optimization complexity; stage two introduces explicit 3D Gaussian primitives with geometric losses to enforce scene consistency. The use of Transformer architectures with Plücker coordinate embeddings effectively captures view-dependent effects and scene structure. This combination allows the model to learn scene geometry and camera poses solely from raw videos, without explicit calibration, representing a significant departure from prior methods relying on SfM, SLAM, or precomputed priors. The framework's modular design and loss functions provide theoretical guarantees for scene consistency and pose accuracy.
Novelty
This research is the first to propose a fully self-supervised, two-stage training framework that does not depend on camera calibration or geometric priors for novel view synthesis. The combination of implicit scene learning via LVSM and explicit geometric alignment with Gaussian primitives introduces a new paradigm for uncalibrated scene reconstruction. Unlike previous methods that either rely on preprocessed data or supervised signals, this approach leverages end-to-end learning directly from raw videos, significantly expanding the scope of scalable 3D scene understanding. Its innovative use of Transformer-based architectures and Plücker embeddings further distinguishes it from existing neural rendering techniques.
Limitations
- Despite its strengths, the method may struggle with scenes exhibiting extreme sparsity of views or rapid camera motion, where implicit representations may fail to capture complex geometry accurately. Computational cost remains high due to Transformer architectures, limiting real-time applications. Additionally, the model's performance can degrade under severe lighting changes or heavy occlusions, indicating a need for more robust feature encoding. Future work should focus on improving efficiency, robustness, and extending to dynamic scenes with temporal consistency.
Future Work
Future directions include integrating multi-scale geometric constraints, enabling dynamic scene modeling, and improving real-time inference capabilities. Exploring more efficient architectures, such as lightweight transformers or neural approximations, could reduce computational costs. Incorporating temporal consistency constraints may extend the framework to video sequences with motion. Additionally, combining multi-modal data, such as depth sensors or semantic cues, could further enhance scene understanding. These advancements aim to make the approach applicable to real-world applications like autonomous navigation, AR/VR, and large-scale scene reconstruction in unconstrained environments.
AI Executive Summary
The rapid development of virtual and augmented reality technologies has heightened the demand for high-fidelity scene reconstruction and novel view synthesis. Traditional methods heavily depend on calibrated cameras and geometric priors, which pose significant limitations in large-scale, unstructured environments. Addressing this challenge, the present work introduces Pensieve, a novel two-stage training framework that enables high-quality view synthesis solely from raw, uncalibrated videos.
In the first stage, the model employs a self-supervised implicit scene reconstruction inspired by LVSM, predicting per-frame latent camera and scene features. This reduces optimization complexity and provides a robust initialization. The second stage introduces explicit geometric alignment by predicting 3D Gaussian primitives and applying Gaussian splatting and depth projection losses, which enforce scene consistency and accurate camera pose estimation. This combination ensures the model learns physically grounded scene structures without relying on precomputed calibration data.
Extensive experiments on datasets like RealEstate10K and DL3DV-140 demonstrate the method’s effectiveness. The model achieves PSNR scores of 26.53 and 21.77 respectively, outperforming prior approaches that depend on calibration or depth supervision. Notably, the approach maintains high rendering quality and precise camera estimation even with minimal input views, thanks to the interpolation-based inference strategy.
This work significantly advances the field by enabling scalable, self-supervised scene understanding from unstructured videos. Its implications span virtual reality, film production, and autonomous systems, where calibration data is often unavailable. While challenges remain in handling extreme scenes or real-time deployment, the proposed framework sets a new benchmark for uncalibrated scene reconstruction and view synthesis, opening pathways for future research in dynamic scenes, efficiency, and multi-modal integration.
Deep Dive
Abstract
Currently almost all state-of-the-art novel view synthesis and reconstruction models rely on calibrated cameras or additional geometric priors for training. These prerequisites significantly limit their applicability to massive uncalibrated data. To alleviate this requirement and unlock the potential for self-supervised training on large-scale uncalibrated videos, we propose a novel two-stage strategy to train a view synthesis model from only raw video frames or multi-view images, without providing camera parameters or other priors. In the first stage, we learn to reconstruct the scene implicitly in a latent space without relying on any explicit 3D representation. Specifically, we predict per-frame latent camera and scene context features, and employ a view synthesis model as a proxy for explicit rendering. This pretraining stage substantially reduces the optimization complexity and encourages the network to learn the underlying 3D consistency in a self-supervised manner. The learned latent camera and implicit scene representation have a large gap compared with the real 3D world. To reduce this gap, we introduce the second stage training by explicitly predicting 3D Gaussian primitives. We additionally apply explicit Gaussian Splatting rendering loss and depth projection loss to align the learned latent representations with physically grounded 3D geometry. In this way, Stage 1 provides a strong initialization and Stage 2 enforces 3D consistency - the two stages are complementary and mutually beneficial. Extensive experiments demonstrate the effectiveness of our approach, achieving high-quality novel view synthesis and accurate camera pose estimation, compared to methods that employ supervision with calibration, pose, or depth information. The code is available at https://github.com/Dwawayu/Pensieve.