Learning to Render Novel Views from Wide-Baseline Stereo Pairs
Proposes a multi-view transformer-based method with epipolar sampling for high-quality novel view synthesis from a single wide-baseline stereo pair, outperforming prior sparse observation approaches.
Key Findings
Methodology
This paper introduces a novel framework combining a multi-view transformer encoder with an image-space epipolar line sampling scheme. The encoder leverages camera pose embeddings and self-attention to produce pixel-aligned scene features, addressing the challenge of sparse observations where many 3D points are only seen once. The epipolar sampling efficiently extracts features along the maximum informative lines, reducing computational costs compared to volumetric rendering. A lightweight cross-attention decoder maps these features into pixel colors for novel views. The training incorporates multi-view regularization and data augmentation, enabling large-scale scene learning without explicit scene optimization.
Key Results
- On RealEstate10k and ACID datasets, our method achieves LPIPS of 0.262, PSNR of 21.38, surpassing PixelNeRF, IBRNet, and GPNR by significant margins. It reconstructs scene geometry more accurately in complex indoor and outdoor environments, with rendering speed over three times faster than traditional volume rendering. The approach effectively handles scenes with minimal overlap, producing multi-view consistent novel views even with sparse input data.
- Ablation studies confirm the importance of multi-view encoding, epipolar sampling, and cross-attention modules. Results demonstrate that the proposed sampling strategy and lightweight decoder are critical for balancing quality and efficiency, enabling real-world large-scale scene synthesis.
- This work marks a step forward in neural scene reconstruction, enabling high-fidelity, multi-view consistent rendering from minimal input, with broad implications for virtual reality, AR, and content creation industries.
Significance
This research addresses the longstanding challenge of synthesizing novel views from extremely sparse data, specifically a single wide-baseline stereo pair. By integrating transformer-based scene encoding with epipolar sampling, it overcomes the limitations of traditional multi-view stereo and volumetric rendering, which are computationally expensive and often inaccurate under sparse observations. The method's ability to produce high-quality, multi-view consistent images efficiently opens new avenues for scalable 3D scene understanding and rendering. Its impact extends to industries requiring real-time scene reconstruction, such as VR/AR, gaming, and film production, providing a practical solution for large-scale, dynamic scene synthesis with minimal input data.
Technical Contribution
The core technical innovation lies in the multi-view transformer encoder that explicitly incorporates camera pose information, enabling pixel-aligned feature extraction across views. The epipolar line sampling scheme optimally exploits the maximum available information by sampling along epipolar lines rather than in 3D space, reducing sampling redundancy. The lightweight cross-attention decoder efficiently maps sparse features to pixel colors, avoiding the high computational costs associated with volumetric rendering. These components collectively facilitate fast, accurate, and scalable scene reconstruction from minimal observations, representing a significant departure from prior methods relying on dense multi-view inputs or scene-specific optimization.
Novelty
This work is the first to successfully apply a multi-view transformer with epipolar sampling to the problem of novel view synthesis from a single wide-baseline stereo pair. Unlike prior approaches that depend heavily on dense multi-view data or expensive volumetric rendering, it leverages sparse, pixel-aligned features and efficient sampling strategies. The integration of a transformer-based scene encoder with a novel sampling scheme and a lightweight renderer constitutes a fundamental innovation, enabling high-quality scene reconstruction without scene-specific optimization. This approach bridges the gap between sparse observations and photorealistic rendering, setting a new standard in neural view synthesis.
Limitations
- The method's performance diminishes in scenes with extreme occlusion or highly dynamic elements, as the scene priors may not fully capture such complexities.
- Dependence on accurate camera pose estimation can affect the quality of reconstructions if pose errors are significant.
- While faster than volumetric methods, real-time rendering in ultra-large scenes remains challenging and requires further optimization.
Future Work
Future research will explore unsupervised scene prior learning to reduce reliance on precise camera parameters, extend the framework to dynamic scenes with temporal consistency, and optimize the architecture for real-time applications. Additionally, integrating learned priors with multi-view stereo to improve geometric accuracy in highly occluded regions is a promising direction.
AI Executive Summary
The quest for realistic, high-fidelity scene rendering from minimal input data has long challenged computer vision and graphics communities. Traditional approaches rely on dense multi-view image collections and explicit scene geometry, which are costly and impractical for large-scale or dynamic scenes. Recent advances in neural rendering, such as Neural Radiance Fields (NeRF), have demonstrated impressive results but typically require hundreds of images per scene, limiting scalability.
This paper introduces a groundbreaking method that synthesizes novel views from just a single wide-baseline stereo image pair. The core innovation is a multi-view transformer encoder that encodes pixel-aligned features by leveraging camera pose information, combined with an image-space epipolar line sampling scheme. Unlike volumetric rendering, which samples densely along rays in 3D space, the proposed approach samples features along epipolar lines in 2D, maximizing the information extracted from sparse observations. The lightweight cross-attention decoder then maps these features into pixel colors, enabling fast and high-quality rendering.
Extensive experiments on large real-world datasets, RealEstate10k and ACID, demonstrate that this approach outperforms existing methods like PixelNeRF, IBRNet, and GPNR across multiple metrics, including LPIPS, PSNR, and SSIM. The method achieves a 3x speed-up over traditional volume rendering and maintains multi-view consistency even with minimal overlap between input images. Ablation studies confirm the importance of each component, from the multi-view encoder to the epipolar sampling strategy.
This work significantly advances the field of neural scene reconstruction, making high-quality, multi-view consistent rendering feasible with sparse input data. Its implications extend to real-time virtual reality, AR applications, and large-scale scene modeling, promising a future where immersive environments can be generated efficiently from minimal observations. Despite these advances, challenges remain in handling extreme occlusions and dynamic scenes, guiding future research directions toward unsupervised learning and real-time optimization.
Deep Dive
Abstract
We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. In this challenging regime, 3D scene points are regularly observed only once, requiring prior-based reconstruction of scene geometry and appearance. We find that existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry and due to the high cost of differentiable rendering that precludes their scaling to large-scale training. We take a step towards resolving these shortcomings by formulating a multi-view transformer encoder, proposing an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray, and a lightweight cross-attention-based renderer. Our contributions enable training of our method on a large-scale real-world dataset of indoor and outdoor scenes. We demonstrate that our method learns powerful multi-view geometry priors while reducing the rendering time. We conduct extensive comparisons on held-out test scenes across two real-world datasets, significantly outperforming prior work on novel view synthesis from sparse image observations and achieving multi-view-consistent novel view synthesis.