Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation
Generative Multiview Relighting combines diffusion harmonization with NeRF-Casting, reaching 31.34 PSNR on Objaverse under extreme lighting variation.
Key Findings
Methodology
The pipeline has two stages. A multiview latent diffusion model jointly relights 64 differently illuminated images to the lighting of a reference image I1. A NeRF-Casting radiance field then reconstructs geometry and appearance. Its per-image shading embedding feeds a three-layer, 128-hidden-unit MLP that slightly perturbs surface normals, absorbing residual highlight misalignment without allowing unconstrained appearance codes to explain away specular structure.
Key Results
- On standard Objaverse assets, Ours achieves PSNR 31.34, SSIM 0.966, and LPIPS 0.053, outperforming NeROIC at 26.13/0.935/0.088, NeRF-Casting+AE at 27.53/0.941/0.067, and IllumiNeRF with true target lighting at 29.22/0.958/0.057.
- On perfectly reflective Shiny Assets, Ours reaches 26.54/0.911/0.090 versus NeROIC's 22.14/0.880/0.113. On 18 real NAVI scenes, it obtains 25.55/0.929/0.060, exceeding NeROIC's 24.01/0.918/0.079.
- Frame-count ablation shows the value of joint inference: standard-asset PSNR rises from 28.39 with one frame to 31.34 with 64, while shiny-asset PSNR rises from 23.85 to 26.54.
Significance
The work addresses a persistent failure mode in neural rendering: uncontrolled illumination violates the static-appearance assumption of standard NeRF systems. It is especially important for metals, glossy plastics, mirrors, and other objects whose appearance changes sharply with lighting and viewing direction. Academically, it connects generative relighting, multiview reasoning, and high-frequency radiance-field rendering. Industrially, it reduces the need for controlled studios and makes internet photographs, mobile capture, and cultural-heritage digitization more usable.
Technical Contribution
The main contributions are threefold. First, the method jointly relights an image set rather than independently sampling each view; 3D self-attention shares object information and pose cross-attention consumes raymap camera representations. Second, shading embeddings modify per-image normals inside NeRF-Casting, unlike appearance embeddings that can absorb genuine view-dependent effects. Third, training uses roughly 300K objects, about 700 Poly Haven environment maps, and additional perfect-mirror renderings, improving recovery of high-frequency reflections.
Novelty
Relative to NeRF-W, NeROIC, IllumiNeRF, and Neural Gaffer, this work introduces multiview generative relighting as a front end for reconstruction from severely non-constant illumination. It also explicitly models residual normal errors after relighting. The fundamental novelty is not diffusion alone, but the combination of cross-view generative consistency with NeRF-Casting's secondary-ray representation of specular appearance.
Limitations
- Training is entirely synthetic, so transparent, translucent, dynamic, heavily occluded, or cluttered real scenes may lie outside the learned distribution and produce incorrect geometry or hallucinated appearance.
- The system assumes known camera poses and many views. Diffusion inference is expensive, while radiance-field optimization takes roughly 30 minutes on 16 NVIDIA A100 GPUs.
- Shading embeddings mainly correct small highlight warps. They cannot repair major geometric mistakes or guarantee physically consistent materials when the diffusion model produces a wrong interpretation.
Future Work
Promising directions include efficient few-view and arbitrary-frame inference, training on real videos, and explicit treatment of transparency, dynamic illumination, and moving objects. Combining diffusion priors with editable physical materials, reliability estimates, and user constraints could suppress hallucinated details. Distillation, latent compression, and pose optimization may eventually enable interactive or mobile deployment.
AI Executive Summary
Reconstructing a 3D object from photographs becomes difficult when every image is captured under a different light. A highlight may move, a shadow may disappear, and a glossy surface may look like a different material. NeRF-W-style appearance embeddings often explain away these effects and produce diffuse models, while inverse-rendering systems such as NeROIC struggle with the ambiguity between lighting, geometry, and material. The paper proposes Generative Multiview Relighting to make the problem more tractable.
The method first uses a latent diffusion model, initialized from a Stable Diffusion 1.5-like architecture, to relight an entire posed image set jointly. Every view is transformed to match the illumination of reference image I1. 3D self-attention exchanges information across views, while raymap pose encodings and cross-attention preserve camera geometry. The harmonized images are then reconstructed with NeRF-Casting. A per-image shading embedding slightly changes the normals used for shading, correcting residual highlight warps without introducing unstable appearance codes.
Results are strong. On standard Objaverse assets, the method reaches 31.34 PSNR, 0.966 SSIM, and 0.053 LPIPS, surpassing IllumiNeRF even when that baseline receives the ground-truth environment map. On perfectly reflective assets, it reaches 26.54/0.911/0.090, compared with NeROIC's 22.14/0.880/0.113. On 18 real NAVI scenes, it obtains 25.55/0.929/0.060. Jointly processing 64 frames is substantially better than one frame. The remaining challenges are synthetic training, known-pose requirements, high computation, and possible generative hallucinations, but the work offers a compelling route to high-fidelity reconstruction from uncontrolled photographs.
Deep Analysis
Background
NeRF represents density and view-dependent color and works well under fixed lighting. NeRF-W introduced per-image appearance embeddings for internet photos, while NeROIC and related inverse-rendering methods estimate materials, normals, and illumination. These approaches can confuse view-dependent highlights with image-specific variation. NeRF-Casting improves specular rendering through secondary rays, but generally assumes constant illumination.
Core Problem
Images captured under different environments vary simultaneously in color, shadow, and reflection. A single image cannot reliably disentangle geometry, material, and lighting, while independently relighting views with diffusion can produce mutually incompatible explanations. Direct NeRF training therefore tends to yield blurry, overly diffuse, or temporally flickering novel views.
Innovation
- ��Joint multiview relighting: all images share one object interpretation instead of being sampled independently.
- ��Image-only reference lighting: the method needs a reference image, not an environment map.
- ��Shading embeddings: per-image normal adjustments target residual highlight errors.
- ��Specular-aware training: mirror versions of roughly 300K objects improve high-frequency reflection recovery.
Methodology
- ��Input: N posed images Ii with arbitrary, different illumination.
- ��Diffusion: each 512×512 image becomes a 64×64×8 latent; clean latents, raymap poses, and a binary reference map condition joint denoising through 3D self-attention and pose cross-attention.
- ��Training: 8, 16, 32, and 64 frames are trained for 200K, 100K, 50K, and 50K steps; CFG is 3.
- ��Reconstruction: NeRF-Casting optimizes density, spatial features, and appearance. Normals are ni(x)=normalize(MLP(f(x),vi)), with a three-layer 128-unit MLP.
- ��Output: the reconstructed field renders unseen views under the reference illumination.
Experiments
Objaverse contains eight standard textured assets and 12 shiny assets, with 64 training and 36 test images per scene. Training views use distinct random illuminations; test views use the reference illumination. NAVI evaluation covers 18 real scenes with at least 64 frames. Baselines are NeROIC, NeRF-Casting plus appearance embeddings, and IllumiNeRF. Metrics are PSNR, SSIM, and LPIPS. The relighting model uses about 700 Poly Haven environment maps and processes 64 views at inference.
Results
On standard assets, Ours reaches 31.34 PSNR versus IllumiNeRF's 29.22. On shiny assets, 26.54 exceeds NeROIC's 22.14 and NeRF-Casting+AE's 21.80. On NAVI, Ours obtains 25.55/0.929/0.060 versus NeROIC's 24.01/0.918/0.079. Increasing simultaneous frames from one to 64 consistently improves quality, confirming that cross-view diffusion reduces lighting-material ambiguity.
Applications
The approach fits internet image collections, mobile multi-environment capture, product digital twins, film assets, museum digitization, and robotic perception. It requires reasonably accurate camera poses, a static object, and sufficient viewpoint coverage. Its strongest benefit is preserving reflections and shadows while enabling novel-view rendering under a chosen reference light.
Limitations & Outlook
The synthetic-only training regime may fail on transparency, complex interreflection, dynamic objects, or severe occlusion. Known poses and many views remain important assumptions. Sixty-four-view diffusion and roughly 30-minute reconstruction on 16 A100 GPUs limit interactive use. Shading embeddings provide flexibility rather than physical guarantees, and cannot remove all errors or hallucinated details from the generative relighting stage.
Plain Language Accessible to non-experts
Imagine photographing a shiny product for a 360-degree display. One photo is taken outdoors, another under office lights, and a third beside a colored lamp. A conventional system may treat the changing shadow as paint, erase a real reflection, or create a strange patch that flickers when the product rotates. This paper first gives all photographs to one “master photographer” that sees them together and edits them so they appear to have been taken under the same lamp.
Seeing the whole set matters: different viewpoints help the system recognize that a bright spot is a reflection on one object, not a permanent sticker. After that, it builds the 3D model. If a highlight is still slightly misplaced, the model makes a tiny per-photo adjustment to the reflection direction, like tilting a mirror a little rather than replacing it with plastic.
The result keeps sharp reflections while remaining stable in new views. On standard and mirror-like Objaverse objects it beats several earlier systems, and it also improves real NAVI photographs. It still needs many images, camera information, and powerful computers, but it offers a practical recipe for turning messy photographs into convincing interactive 3D objects.
ELI14 Explained like you're 14
Suppose you want to scan a shiny sword for a video game. One photo is taken in sunlight, one in a classroom, and one beside a red lamp. The sword looks different each time: the bright spot moves and the shadows change. If a computer simply glues the pictures together, it may think the sword has colorful bumps. When the camera moves, the result can flicker like a broken game texture!
This paper uses a super-smart editor that looks at many photos at once. First it changes their lighting so they look as if they were taken under the same lamp. Why together? Because every picture can remind the model, “Hey, this is the same sword from another angle!” That is much safer than editing each picture independently and accidentally inventing different swords.
Next, the computer builds a 3D version. If a shiny spot is a little off, it makes a tiny adjustment to the reflection direction for that picture. It does not repaint the whole sword or remove the shine. That helps preserve metal, glass-like highlights, and shadows.
On Objaverse, the method gets 31.34 PSNR for ordinary objects and 26.54 for very shiny ones, beating older approaches. It still needs accurate camera positions, lots of pictures, and serious GPU power. But imagine scanning game props, toys, or products with your phone and getting a stable rotating model—pretty cool, right?
Glossary
Neural Radiance Field (神经辐射场)
A neural representation of 3D density and appearance that can render images from new camera positions. Technically, it integrates predicted radiance and density along camera rays.
The paper uses a NeRF-Casting radiance field after relighting.
Diffusion Model (扩散模型)
A generative model trained to reverse a gradual noising process. In plain terms, it learns to reconstruct plausible images from corrupted representations.
It jointly relights multiple input views to one reference illumination.
NeRF-Casting
A radiance-field method that traces reflected secondary rays to represent specular appearance. This supports sharp, high-frequency reflections better than low-frequency lighting models.
It supplies the reconstruction backbone in the second stage.
Shading Embedding (着色嵌入)
A vector associated with each training image that slightly changes the normals used for shading. It compensates for small highlight shifts left by diffusion relighting.
It replaces unconstrained appearance embeddings in the proposed reconstruction design.
View-dependent Appearance (视角相关外观)
Appearance that changes with viewing direction, such as metallic reflections or glossy highlights. It is therefore not a fixed texture alone.
Recovering this effect, especially on shiny objects, is the paper's main target.
Open Questions Unanswered questions from this research
- 1 It remains unclear how to maintain consistency with very few real images, inaccurate poses, or unknown camera calibration. Solving this likely requires joint pose estimation and uncertainty-aware generation.
- 2 Diffusion may create visually plausible but physically incorrect details. Future systems need material, geometry, and lighting constraints that distinguish faithful reconstruction from convincing hallucination.
Applications
Immediate Applications
Product and film-asset reconstruction
Studios can capture products or props under uncontrolled lighting, harmonize the images, and render stable novel views. Accurate poses and a mostly static object are required; the method is particularly useful for metal, lacquer, jewelry, and other glossy surfaces.
Cultural heritage and web-object digitization
Museums and data platforms can build rotating 3D assets from photographs collected in different places, reducing controlled-light capture costs. Compared with NeROIC and NeRF-Casting+AE, the method preserves more shadows and specular structure.
Long-term Vision
Low-constraint 3D scanning
With faster inference and weaker pose requirements, phone videos or web images could produce relightable 3D assets for AR, virtual try-on, games, and robotics. Transparency, motion, and real-world lighting diversity remain major obstacles.
Abstract
Reconstructing the geometry and appearance of objects from photographs taken in different environments is difficult as the illumination and therefore the object appearance vary across captured images. This is particularly challenging for more specular objects whose appearance strongly depends on the viewing direction. Some prior approaches model appearance variation across images using a per-image embedding vector, while others use physically-based rendering to recover the materials and per-image illumination. Such approaches fail at faithfully recovering view-dependent appearance given significant variation in input illumination and tend to produce mostly diffuse results. We present an approach that reconstructs objects from images taken under different illuminations by first relighting the images under a single reference illumination with a multiview relighting diffusion model and then reconstructing the object's geometry and appearance with a radiance field architecture that is robust to the small remaining inconsistencies among the relit images. We validate our proposed approach on both synthetic and real datasets and demonstrate that it greatly outperforms existing techniques at reconstructing high-fidelity appearance from images taken under extreme illumination variation. Moreover, our approach is particularly effective at recovering view-dependent "shiny" appearance which cannot be reconstructed by prior methods.