NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
NeuS introduces a bias-free volume rendering approach combined with neural signed distance functions (SDF) for high-fidelity multi-view surface reconstruction, outperforming IDR and NeRF.
Key Findings
Methodology
NeuS employs a neural signed distance function (SDF) as the scene representation, trained via a novel bias-free volume rendering formulation. The core architecture consists of two multilayer perceptrons (MLPs): one predicting the signed distance f(x) and another encoding color c(x, v). The key innovation is the introduction of an S-density distribution φs(f(x)), derived from the SDF, which guides the volume rendering process. Unlike traditional density-based methods, the proposed formulation ensures that the surface corresponds to the zero-level set of the SDF and that the rendering process remains unbiased in the first-order approximation. During training, hierarchical sampling and Eikonal regularization are employed to stabilize the optimization and enforce geometric consistency. The method is evaluated on DTU and BlendedMVS datasets, demonstrating superior surface reconstruction quality, especially in scenes with complex geometry and self-occlusion.
Key Results
- Quantitative results show that NeuS achieves an average Chamfer distance of 0.77 on the DTU dataset without mask supervision, outperforming IDR (1.63) and NeRF (1.54). The method effectively captures fine details and thin structures, with a significant reduction in geometric bias and noise.
- Qualitative reconstructions reveal that NeuS produces cleaner, more accurate surface meshes, especially in challenging regions with abrupt depth changes and occlusions. The reconstructed surfaces exhibit higher fidelity and fewer artifacts compared to baseline methods.
- Ablation studies confirm that the bias-free volume rendering formula significantly improves surface accuracy, reducing reconstruction errors by approximately 30% compared to naive density-based approaches. The hierarchical sampling strategy further enhances convergence and detail preservation.
Significance
This work addresses a fundamental challenge in neural scene representation—accurately extracting high-quality surfaces from implicit functions learned via volume rendering. By eliminating inherent geometric bias, NeuS enables more precise surface reconstructions in complex scenes with self-occlusion and thin structures. Its ability to operate without foreground masks broadens its applicability in real-world scenarios, such as cultural heritage digitization, virtual environment creation, and robotics. Theoretically, it bridges the gap between volumetric neural rendering and explicit surface extraction, setting a new standard for neural implicit surface reconstruction. Practically, it paves the way for more robust, scalable, and detailed 3D scene understanding systems, impacting industries ranging from entertainment to industrial inspection.
Technical Contribution
NeuS's primary technical innovation lies in deriving an unbiased volume rendering formula tailored for neural signed distance functions. This involves defining an S-density distribution φs(f(x)) that peaks at the surface, and constructing a novel weight function w(t) that guarantees maximum contribution at the surface intersection while respecting occlusion relationships. The derivation employs a theoretical analysis of the first-order approximation of SDF, ensuring that the reconstructed surface aligns with the zero-level set. The method integrates hierarchical importance sampling and Eikonal regularization to stabilize training and enforce geometric consistency. These contributions collectively enable the neural network to learn an accurate, high-resolution implicit surface directly from 2D images, without requiring explicit masks or dense supervision.
Novelty
This research is the first to propose a bias-free volume rendering formulation specifically designed for neural signed distance functions, addressing a critical limitation in prior volume rendering-based scene representations like NeRF. Unlike existing methods that rely on density fields prone to geometric bias, NeuS leverages the properties of SDF to define an S-density distribution that naturally encodes the surface as a zero-level set. The novel weight function w(t), derived from this distribution, ensures that the surface intersection contributes maximally to the rendering, eliminating the bias that causes geometric inaccuracies. This approach effectively combines the robustness of volume rendering with the geometric precision of SDF, marking a significant step forward in neural implicit surface reconstruction.
Limitations
- Despite its high accuracy, NeuS requires substantial computational resources and lengthy training times (~14-16 hours), limiting its immediate deployment in real-time applications.
- The method assumes scenes are static and geometrically continuous; scenes with extreme lighting variations, non-manifold geometries, or highly discontinuous surfaces may pose challenges.
- Currently, the approach is primarily designed for static scene reconstruction; extending it to dynamic scenes with temporal changes remains an open research question.
Future Work
Future research will focus on reducing computational costs through model compression and more efficient sampling strategies. Extending NeuS to handle dynamic scenes and non-Lambertian materials is another promising direction. Incorporating physically-based lighting models could further enhance realism. Additionally, integrating multi-scale features and developing real-time variants will broaden practical applications in AR/VR, robotics, and digital heritage preservation.
AI Executive Summary
Reconstructing detailed 3D scenes from multi-view images remains a core challenge in computer vision, especially when aiming for high fidelity in complex environments. Traditional methods such as multi-view stereo (MVS) and point cloud fusion rely heavily on texture richness and accurate correspondence, often struggling with scenes featuring thin structures, severe occlusion, or non-Lambertian surfaces. These limitations hinder the ability to generate complete and precise 3D models necessary for applications like virtual reality, cultural heritage digitization, and industrial inspection.
Recent advances in neural implicit scene representations, exemplified by methods like NeRF, have revolutionized novel view synthesis by employing volume rendering to produce photorealistic images. However, these approaches primarily focus on appearance and lack the explicit geometric constraints needed for high-quality surface extraction. Methods like IDR attempt to bridge this gap by incorporating surface regularization but still suffer from geometric biases introduced by density-based volume rendering, especially in scenes with complex geometry or abrupt depth changes.
Addressing these challenges, the paper introduces NeuS—a novel neural surface reconstruction framework that combines the strengths of neural implicit functions and unbiased volume rendering. Central to NeuS is the use of a signed distance function (SDF) to represent scene geometry, where the surface corresponds to the zero-level set of the SDF. To train this implicit surface from 2D images alone, the authors develop a new volume rendering formulation that is free from the geometric bias inherent in traditional density-based methods. This formulation leverages a specially designed S-density distribution, φs(f(x)), which peaks at the surface, ensuring that the rendered images accurately reflect the scene geometry.
The core technical innovation involves deriving a weight function w(t) based on this S-density, which guarantees that the maximum contribution to the pixel color occurs at the surface intersection. This approach ensures that the reconstructed surface aligns precisely with the zero-level set of the SDF, even in the absence of explicit masks. The training process incorporates hierarchical sampling and Eikonal regularization to stabilize optimization and enforce geometric consistency. Extensive experiments on the DTU and BlendedMVS datasets demonstrate that NeuS outperforms existing methods like IDR and NeRF, especially in scenes with complex structures, self-occlusion, and thin features. Quantitative metrics show a significant reduction in reconstruction error, and qualitative results reveal more accurate and cleaner surface meshes.
This work advances the field by providing a theoretically grounded, bias-free approach to neural surface reconstruction, bridging the gap between volumetric neural rendering and explicit surface extraction. Its ability to operate without foreground masks and handle complex geometries makes it highly promising for real-world applications. Future directions include optimizing computational efficiency, extending to dynamic scenes, and integrating physically-based rendering for enhanced realism. Overall, NeuS marks a pivotal step toward robust, high-fidelity neural scene understanding, with broad implications across industry and academia.
Deep Dive
Abstract
We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.
References (20)
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
Michael Oechsle, Songyou Peng, Andreas Geiger
NeRF++: Analyzing and Improving Neural Radiance Fields
Kai Zhang, Gernot Riegler, Noah Snavely et al.
Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
Lior Yariv, Yoni Kasten, Dror Moran et al.
Pixelwise View Selection for Unstructured Multi-View Stereo
Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm et al.
Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
M. Niemeyer, L. Mescheder, Michael Oechsle et al.
Representing Scenes as Neural Radiance Fields for View Synthesis
Real-Time Visibility-Based Fusion of Depth Maps
Paul C. Merrell, Amir Akbarzadeh, Liang Wang et al.
PatchMatch: a randomized correspondence algorithm for structural image editing
Connelly Barnes, Eli Shechtman, Adam Finkelstein et al.
A Globally Optimal Algorithm for Robust TV-L1 Range Image Integration
C. Zach, T. Pock, H. Bischof
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Tim Salimans, Diederik P. Kingma
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
C. Choy, Danfei Xu, JunYoung Gwak et al.
Poxels: Probabilistic Voxelized Volume Reconstruction
J. Bonet
A Point Set Generation Network for 3D Object Reconstruction from a Single Image
Haoqiang Fan, Hao Su, L. Guibas
Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
Chen-Hsuan Lin, Chen Kong, Simon Lucey
Image-based reconstruction of wire art
Lingjie Liu, Duygu Ceylan, Cheng Lin et al.
Learning a Multi-View Stereo Machine
Abhishek Kar, Christian Häne, Jitendra Malik
Learning Implicit Fields for Generative Shape Modeling
Zhiqin Chen, Hao Zhang
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
Nanyang Wang, Yinda Zhang, Zhuwen Li et al.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
Cited By (20)
NeuDonatello: Uncertainty-Aware Framework for Accurate Neural SDF Learning
ExMesh++: From Multi-View Images to Relightable UV-PBR Mesh Assets via Topology-Adaptive Reconstruction and Decomposition
TopoSurfel: Closing the Loop between Gaussian Surfels and Meshes for Surface Reconstruction
Recent Advances in Image-Based 3D Reconstruction: a Photogrammetric Perspective on Conventional and Learning-Based Techniques
View-adaptive renderer for view-consistent 2D-to-3D generation
Gaussian Sculpting: End-to-End Controllable Surface Reconstruction via Field Optimization
Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition
SuperSDF: Sparse SDF Super-Resolution for Surface Extraction
Sparse-View Surface Reconstruction using Gaussian Splatting through High-Confidence Depth Propagation with Normal Priors
Dual Contouring over Expanded Cubes (DCx) for Zero-Level Set Extraction from Neural Unsigned Distance Functions
Online Neural Space Time Memory for Dynamic Novel View Synthesis
G$^2$SR: Geometric Methods for Fast and Memory-Efficient Gaussian-based Surface Reconstruction
Differentiable Voxelization of Surface Representations
Evaluating 3D Scene Representations for Aerial Photogrammetry across Diverse Cityscapes
FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System
Plenoptic Condensation: A Novel Approach to Generalized Scene Reconstruction
From Distances to Trajectories: Real-Time Signed Distance Function Mapping and Distance-Accelerated Motion Planning for UAVs
Seeing Before Generating: Object Perception Enhances Single-View 3D Reconstruction
Axolotl3D: a Unified Framework for Faithful 3D Shape Completion
Uncertainty-guided neural radiance fields against diverse degradation factors