Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition

Recent advances in neural scene representations like NeRF and NeuS have enabled high-fidelity static scene reconstruction. However, these methods typically embed appearance under fixed lighting, limiting relighting capabilities. Traditional inverse rendering approaches attempt to separate material and illumination but face ambiguity issues, often relying on multi-illumination captures or priors. Dynamic scene reconstruction remains challenging due to the difficulty of disentangling geometry, material, and lighting in moving scenes. The need for methods that leverage motion to improve inverse rendering accuracy is increasingly urgent, especially for AR/VR and digital content applications.

The core challenge lies in the ill-posed nature of inverse rendering: multiple material and lighting combinations can produce identical observed radiance. Static multi-view captures provide limited constraints, making it difficult to accurately decompose scene properties. Incorporating object motion introduces diverse light-surface interactions, offering richer constraints. However, accurately tracking and modeling dynamic scenes with complex geometries and materials remains difficult, especially under noisy conditions. Overcoming these hurdles is essential for realistic scene editing and virtual content creation.

The paper's key innovations include: 1) a three-stage pipeline combining NeuS, 3D Gaussian models, and physically-based rendering; 2) leveraging object motion to introduce diverse lighting interactions, strengthening inverse problem constraints; 3) integrating ray tracing for occlusion and global illumination modeling; 4) joint optimization of geometry, pose, material, and lighting parameters. These innovations enable high-precision, robust material-light disentanglement in dynamic scenes, surpassing static or single-stage methods. The approach uniquely exploits motion cues, filling a critical gap in current inverse rendering research.

• �� Stage 1: Use NeuS to estimate coarse geometry and initial poses, guided by 2D feature matching, minimizing rendering, eikonal, and mask losses. • Stage 2: Switch to 3D Gaussian models, refining geometry and pose jointly, utilizing ray tracing for occlusion and depth accuracy. • Stage 3: Add material parameters and environment maps, optimize via differentiable rendering, modeling BRDFs with diffuse and specular components. • Throughout, iterative optimization refines parameters, leveraging object motion to provide multiple lighting conditions per surface point, thus improving material-light separation.

Experiments utilize synthetic HOT3D datasets with ground-truth materials and relit views, simulating static, turntable, and hand-held scenarios. Metrics include material error, lighting error, and geometric accuracy. Comparisons with static methods demonstrate significant improvements in material and illumination estimation under dynamic conditions. Real-world videos of hand-held objects further validate robustness. Ablation studies analyze each pipeline component's contribution, confirming the importance of motion-based constraints. Results show consistent performance gains across diverse scenarios.

Quantitative analysis reveals a 25% reduction in material error and a 20% decrease in lighting error in synthetic datasets under motion. In real videos, the pipeline maintains high accuracy despite noise. The method outperforms static approaches, especially in complex lighting environments, confirming the advantage of leveraging object motion. Fine geometric details are captured more accurately, and material separation is clearer, enabling realistic relighting and editing. These results demonstrate the approach's potential for practical applications.

The technique can be directly applied to virtual try-on, AR scene editing, and digital content creation, where capturing a short video of a moving object suffices for high-quality material and lighting reconstruction. It requires minimal user input, making it accessible for industry use. Long-term, the method could enable real-time scene understanding, dynamic scene editing, and immersive virtual environments, transforming how digital content is generated and manipulated in entertainment, design, and e-commerce sectors.

Dependence on accurate pose tracking means errors can propagate, especially in fast or blurry motions. The current model supports primarily reflective materials, with limited support for translucent or subsurface scattering surfaces. Computational complexity remains high, particularly during ray tracing and high-resolution environment map optimization, limiting real-time deployment. Future work must address these issues by improving tracking robustness, expanding material models, and optimizing algorithms for efficiency.