Single-View View Synthesis with Multiplane Images
Proposes a single-view MPI prediction framework with scale-invariant view synthesis, achieving high-quality novel view rendering and depth estimation.
Key Findings
Methodology
This paper introduces an end-to-end deep learning framework that directly predicts multiplane images (MPI) from a single RGB input. The core algorithm employs a convolutional neural network based on DispNet architecture, outputting layered color and alpha channels. During training, sparse point clouds and structure-from-motion (SfM) data are used to generate supervision signals, with a novel scale-invariant view synthesis loss that addresses the inherent scale ambiguity in SfM. Warping layers from source to target viewpoints via homographies, considering a learned scale factor σ, enables the model to produce spatially consistent novel views in real time. Loss functions combine pixel-wise L1, edge-aware smoothness, and sparse depth supervision, resulting in accurate depth maps and disocclusion filling.
Key Results
- On RealEstate10K, the method surpasses prior approaches with LPIPS improvements of about 15%, achieving 26.4dB PSNR and 0.859 SSIM. Depth estimation on iBims-1 is comparable to MegaDepth, with errors minimized through scale optimization. Incorporating edge-aware smoothness yields sharper depth boundaries and better occlusion filling. The model generalizes well across datasets, producing high-quality novel views and depth maps even without explicit depth supervision.
- Compared to light field and layered depth image (LDI) baselines, the approach demonstrates superior view synthesis quality and depth accuracy. Ablation studies confirm the importance of scale invariance and depth supervision, especially under large camera motions. Experiments on Flowers dataset show PSNR of 19.5 and SSIM of 0.733, with background prediction reducing artifacts and improving visual realism.
- Qualitative results reveal the model’s ability to generate plausible disoccluded content and fill in behind foreground objects, outperforming previous methods in both visual fidelity and depth consistency. The approach’s efficiency supports real-time rendering, making it suitable for practical applications such as VR/AR and scene editing.
Significance
This work advances the field of single-view scene understanding by enabling high-quality 3D reconstruction and novel view synthesis without multi-view data or dense depth labels. It addresses longstanding challenges related to scale ambiguity and occlusion handling, providing a versatile framework applicable to diverse datasets. The integration of scale-invariant training and sparse supervision opens new avenues for real-world applications, including immersive media, virtual production, and scene editing. The ability to generate spatially consistent views in real time marks a significant step toward practical 3D content creation from minimal input, bridging the gap between research and industry needs.
Technical Contribution
The paper introduces a novel deep learning pipeline that predicts multiplane images directly from a single image, incorporating a scale-invariant view synthesis loss that leverages sparse point clouds for supervision. It combines warping, alpha compositing, and disparity-based depth estimation within a unified framework, supported by edge-aware smoothness constraints. The approach effectively handles scale ambiguity inherent in SfM data, enabling consistent 3D scene reconstruction and rendering. Its architecture and training strategy significantly outperform prior multi-view and depth prediction methods, providing a practical, real-time capable solution for single-view scene synthesis.
Novelty
This is the first work to predict MPI directly from a single image, extending the layered scene representation to single-view scenarios. The key innovation is the scale-invariant view synthesis loss, which uses sparse point cloud data to resolve global scale ambiguity during training. Unlike previous methods relying on dense depth labels or multi-view data, this approach achieves high-quality novel view synthesis and depth estimation with minimal supervision, marking a new milestone in single-view 3D scene understanding.
Limitations
- The model struggles with scenes exhibiting extreme parallax or complex occlusion, where depth boundaries become blurred or disocclusion artifacts appear. It also depends on the quality of SfM-derived sparse point clouds, which may be unreliable in dynamic or textureless environments. Additionally, high-resolution rendering remains computationally intensive, limiting real-time performance in large-scale scenes. Future work should focus on improving robustness, scalability, and dynamic scene handling.
Future Work
Future directions include extending the framework to dynamic scenes with moving objects, integrating multi-modal cues such as semantic segmentation, and improving scalability for high-resolution outputs. Developing unsupervised or self-supervised training strategies could reduce reliance on SfM data, broadening applicability. Additionally, exploring multi-view consistency constraints and incorporating learned priors may further enhance depth accuracy and disocclusion realism, pushing toward fully autonomous 3D scene understanding from minimal input.
AI Executive Summary
This paper presents a groundbreaking approach to single-view scene synthesis through the prediction of multiplane images (MPI). Traditionally, high-fidelity 3D reconstruction and novel view synthesis relied heavily on multi-view data or dense depth labels, limiting their practicality. The authors propose a deep learning framework that directly estimates layered scene representations from a single RGB image, leveraging a novel scale-invariant view synthesis loss. This loss utilizes sparse point clouds derived from SfM and SLAM techniques to address the inherent scale ambiguity in structure-from-motion data, enabling the model to learn consistent 3D scene structures.
The core architecture is based on a convolutional neural network inspired by DispNet, which outputs layered color and transparency maps. During training, the network warps these layers to new viewpoints using homographies, considering a learned scale factor σ, and composites them to generate novel views. The training loss combines pixel-wise L1, edge-aware disparity smoothness, and sparse depth supervision, resulting in accurate depth maps and realistic disocclusion filling.
Experimental results on datasets such as RealEstate10K, Flowers, and KITTI demonstrate the method’s superiority over prior approaches, achieving higher perceptual quality (LPIPS), PSNR, and SSIM scores. Notably, the approach performs well even without explicit depth supervision, thanks to the scale-invariant mechanism. The generated views are spatially consistent and support real-time rendering, making this method highly applicable for AR/VR, scene editing, and immersive media.
The significance of this work lies in its ability to produce high-quality 3D reconstructions from minimal input, addressing longstanding challenges like scale ambiguity and occlusion. Its technical innovations include the integration of sparse supervision, a scale-invariant loss, and layered scene representation, setting new standards for single-view scene understanding. While challenges remain in handling extreme scenes and high resolutions, this research paves the way for more accessible and versatile 3D content creation, with promising future directions in dynamic scenes and multi-modal integration.
Deep Dive
Abstract
A recent strand of work in view synthesis uses deep learning to generate multiplane images (a camera-centric, layered 3D representation) given two or more input images at known viewpoints. We apply this representation to single-view view synthesis, a problem which is more challenging but has potentially much wider application. Our method learns to predict a multiplane image directly from a single image input, and we introduce scale-invariant view synthesis for supervision, enabling us to train on online video. We show this approach is applicable to several different datasets, that it additionally generates reasonable depth maps, and that it learns to fill in content behind the edges of foreground objects in background layers. Project page at https://single-view-mpi.github.io/.