Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
Proposes Scene Representation Networks (SRNs), a continuous, differentiable scene model trained end-to-end from 2D images, capturing geometry and appearance without supervision.
Key Findings
Methodology
SRNs model scenes as continuous functions Φ: R3→Rn, mapping spatial coordinates to local features. Using a differentiable ray marching algorithm with an LSTM to predict step sizes, the model finds scene intersections along camera rays. The renderer Θ converts scene features into images based on camera parameters, trained solely on posed 2D images. The approach enforces multi-view consistency, learns geometry and appearance priors, and generalizes across scenes. It supports novel view synthesis, few-shot reconstruction, and non-rigid face modeling.
Key Results
- On ShapeNet, SRNs achieved PSNR 26.23dB for novel view synthesis, outperforming baselines like dGQN (20.85dB). In few-shot tasks, single-view PSNR was 18.11dB, and multi-view interpolation showed smooth geometry and appearance transitions. The model effectively captures complex deformations and generalizes to unseen viewpoints.
- In Shepard-Metzler objects and face animation, SRNs demonstrated continuous geometry and appearance interpolation, successfully modeling non-rigid deformations without explicit shape supervision.
- The model's strong generalization allows high-quality multi-view consistent rendering across different scenes, validating its potential for scene understanding and synthesis in diverse applications.
Significance
This work addresses the reliance on explicit 3D supervision in neural scene representations by introducing a continuous, structure-aware model trained solely from 2D images. It advances the field by enabling high-resolution, multi-view consistent scene synthesis, with broad implications for virtual reality, robotics, and content creation. The ability to learn geometry and appearance priors across scenes reduces data requirements and enhances generalization, marking a significant step toward unsupervised 3D scene understanding.
Technical Contribution
SRNs integrate a continuous scene function Φ with a differentiable ray marching algorithm, enabling end-to-end training without depth labels. The learned LSTM predicts adaptive step sizes for efficient intersection search, improving geometric accuracy. The use of a hypernetwork Ψ for multi-scene modeling allows shared renderer parameters and scene-specific scene parameters, facilitating cross-scene generalization. These innovations collectively push the boundaries of neural scene representations, combining geometric fidelity with high-resolution appearance modeling.
Novelty
This is the first work to combine continuous scene functions with a learnable, differentiable ray marching process, enabling unsupervised, high-resolution multi-view synthesis. Unlike prior discrete or weakly structured models, SRNs directly encode scene geometry and appearance in a continuous, differentiable framework, providing theoretical guarantees of multi-view consistency and enabling generalization across scenes. The integration of a hypernetwork for multi-scene learning further distinguishes this approach.
Limitations
- The current model does not explicitly model view-dependent effects or translucency, limiting realism in scenes with complex lighting or transparent objects.
- Training is computationally intensive, requiring significant GPU resources and time (~6 days), which may hinder scalability.
- Shape and appearance are entangled, making it challenging to independently control or edit these aspects. Future work should address these limitations by incorporating physical lighting models and disentanglement techniques.
Future Work
Future directions include modeling view-dependent effects, incorporating physically-based rendering, and probabilistic scene representations to handle uncertainty. Extending the framework to larger scenes and real-world data, improving computational efficiency, and enabling explicit control over shape and appearance are key goals. Additionally, integrating physical priors and multi-modal data could further enhance scene realism and robustness.
AI Executive Summary
Understanding and reconstructing 3D scenes from 2D images remains a fundamental challenge in computer vision and graphics. Traditional methods rely on explicit geometric supervision, such as depth labels or mesh annotations, which are costly and limited in generalization. Recent neural approaches, like NeRF and GQN, have made strides in synthesizing novel views but often depend on supervised data or discrete scene representations that lack continuous spatial fidelity.
This paper introduces Scene Representation Networks (SRNs), a novel framework that models scenes as continuous, differentiable functions. The core idea is to represent a scene as a function Φ: R3→Rn, mapping each spatial coordinate to a local feature vector. To render images from arbitrary viewpoints, SRNs employ a differentiable ray marching algorithm, where a learned LSTM predicts adaptive step sizes along camera rays to find scene intersections. The scene features are then mapped to pixel colors by a simple per-pixel MLP, ensuring multi-view consistency and high-resolution rendering.
Training SRNs involves only posed 2D images and camera parameters, without requiring depth or shape supervision. The model learns geometric and appearance priors across scenes via a hypernetwork Ψ, which maps latent codes to scene parameters. Extensive experiments on ShapeNet and Shepard-Metzler datasets demonstrate that SRNs outperform baselines like dGQN, achieving PSNR scores of 26.23dB and robustly generalizing to unseen views and scenes. The model captures complex non-rigid deformations, supports latent space interpolation, and can perform few-shot reconstruction, highlighting its versatility.
This approach significantly advances neural scene understanding by combining continuous scene functions with differentiable rendering, enabling high-quality, multi-view consistent synthesis in a fully unsupervised manner. Its ability to generalize across scenes and handle complex geometries opens new avenues for applications in virtual reality, robotics, and digital content creation. Despite current limitations in modeling view-dependent effects and computational costs, future work aims to incorporate physical effects and improve efficiency, promising a more realistic and scalable scene representation framework.
Deep Dive
Abstract
Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.