RNG: Relightable Neural Gaussians
RNG uses neural Gaussian points to achieve fast relightable 3D reconstruction, supporting complex materials and soft boundaries, with only 1.3 hours training.
Key Findings
Methodology
This paper introduces the Relightable Neural Gaussians (RNG) framework based on 3D Gaussian splatting, where each Gaussian point carries a latent feature vector conditioned on view and light directions. The model implicitly learns material and reflectance properties without relying on explicit shading models. A two-stage optimization process is employed: the first stage captures geometry via forward shading, while the second incorporates shadow cues and a depth refinement network to improve shadow accuracy. A hybrid forward-deferred rendering strategy balances geometric detail and shadow realism, enabling rapid training (1.3 hours) and real-time rendering at 60 fps. The neural Gaussian decoder, an MLP, interprets the latent features conditioned on view, light, and shadow cues, producing high-fidelity relighting results.
Key Results
- Extensive experiments on real and synthetic datasets demonstrate that RNG surpasses NeRF-based NRHints and 3DGS in detail fidelity, shadow sharpness, and efficiency. PSNR scores reach 28.4-38.5, LPIPS drops to 0.014-0.043, with training times reduced by over 20× compared to NRHints. Shadow edges are sharper and more accurate, especially in soft boundary and complex material scenes. The method achieves real-time rendering at 60 fps, significantly improving practical usability.
- Shadow quality outperforms GS3, with crisper edges and richer details. The shadow cue and depth refinement effectively mitigate shadow blurring and depth inaccuracies, resulting in more realistic scene reconstructions.
- Ablation studies confirm that shadow cues, depth refinement, and hybrid optimization are critical for balancing geometric detail and shadow realism, with each component contributing to overall performance improvements.
Significance
This work advances the field of neural scene reconstruction by enabling fast, high-quality relighting of objects with complex materials and soft boundaries. It addresses longstanding challenges in decomposing light, geometry, and material, especially for non-smooth surfaces, without relying on restrictive analytical shading assumptions. The approach significantly reduces training time and enhances shadow realism, making it suitable for real-world applications such as virtual reality, digital content creation, and industrial design. By supporting arbitrary view and light directions, RNG paves the way for more flexible and scalable scene synthesis, bridging the gap between research and practical deployment.
Technical Contribution
The core technical innovation lies in the implicit modeling of reflectance via neural latent vectors attached to each Gaussian point, conditioned on view and light directions. The shadow cue, derived from shadow mapping, and the depth refinement network, improve shadow sharpness and accuracy. The hybrid optimization strategy combines the strengths of forward and deferred shading, balancing geometry fidelity and shadow realism. This framework enables efficient training and real-time rendering, surpassing previous methods in detail and shadow quality, and broadening the applicability of neural scene representations.
Novelty
This is the first work to leverage neural Gaussian point clouds for relightable scene reconstruction that supports soft boundaries and complex materials without explicit shading models. Unlike prior methods relying on surface constraints or analytical BRDFs, RNG employs a flexible, learned implicit reflectance model conditioned on view and light, combined with shadow cues and depth refinement. This approach achieves superior detail, shadow sharpness, and efficiency, representing a significant step forward in neural scene synthesis.
Limitations
- Currently, the method primarily supports point light sources; integration of environment lighting and global illumination remains limited, restricting scene realism in outdoor or complex lighting conditions.
- The depth refinement network may struggle with highly occluded or geometrically complex scenes, affecting shadow accuracy and detail fidelity.
- Generalization to large-scale or highly dynamic scenes needs further validation, and computational costs, though reduced, still pose challenges for real-time applications in very complex environments.
Future Work
Future directions include extending the framework to handle environment lighting and global illumination, improving depth refinement robustness, and enabling dynamic scene reconstruction. Incorporating real-time updates for moving objects and scalable scene representations will further broaden practical applications in AR/VR, gaming, and industrial visualization.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭，锅里的汤代表场景，各种火候和调料代表不同的光线和材质。传统做法就像用固定的食谱，只能做出一种味道。而科学家们发明了一种神奇的厨具，可以根据不同的火候和调料，快速调整汤的颜色和味道，还能让汤在不同灯光下看起来都很漂亮。这就像RNG技术，用很多小点点组成的“神经高斯点云”来代表场景的每一部分，每个点都记着它的颜色和材质信息。它还能根据不同的光线和角度，调整阴影和反射，让场景看起来更真实。最厉害的是，这个方法只需要很短的时间就能学会一幅场景，然后每秒可以渲染出60帧画面，就像动画一样流畅。这样，未来的虚拟世界会变得更快、更漂亮，也更容易让人沉浸其中。
ELI14 Explained like you're 14
想象你在玩一个超级逼真的视频游戏，里面的场景和角色都像真实生活一样，但制作这些场景需要花费很多时间和努力。科学家们发明了一种叫RNG的方法，就像用一种神奇的画笔，可以快速画出各种复杂的场景，而且还能让场景在不同的灯光下看起来都很自然。这个方法用很多小点点组成的“神经高斯点云”来代表场景的每一部分，每个点都记着它的颜色和材质信息。它还能根据不同的光线和角度，调整阴影和反射，让场景看起来更真实。最酷的是，这个方法只需要很短的时间就能学会一幅场景，然后每秒可以渲染出60帧画面，就像动画一样流畅。未来，我们可以用它做出更复杂、更逼真的虚拟世界，甚至让虚拟场景变得和真实一样精彩！
Abstract
3D Gaussian Splatting (3DGS) has shown impressive results for the novel view synthesis task, where lighting is assumed to be fixed. However, creating relightable 3D assets, especially for objects with ill-defined shapes (fur, fabric, etc.), remains a challenging task. The decomposition between light, geometry, and material is ambiguous, especially if either smooth surface assumptions or surfacebased analytical shading models do not apply. We propose Relightable Neural Gaussians (RNG), a novel 3DGS-based framework that enables the relighting of objects with both hard surfaces or soft boundaries, while avoiding assumptions on the shading model. We condition the radiance at each point on both view and light directions. We also introduce a shadow cue, as well as a depth refinement network to improve shadow accuracy. Finally, we propose a hybrid forward-deferred fitting strategy to balance geometry and appearance quality. Our method achieves significantly faster training (1.3 hours) and rendering (60 frames per second) compared to a prior method based on neural radiance fields and produces higher-quality shadows than a concurrent 3DGS-based method. Project page: https://www.whois-jiahui.fun/project_pages/RNG.