ExMesh++: From Multi-View Images to Relightable UV-PBR Mesh Assets via Topology-Adaptive Reconstruction and Decomposition
ExMesh++ employs topology-adaptive explicit mesh reconstruction and UV-PBR decomposition, achieving high-accuracy geometry and relightable assets from multi-view images.
Key Findings
Methodology
ExMesh++ introduces a two-stage pipeline: first, it performs topology-adaptive mesh refinement via vertex splitting and merging, maintaining UV consistency to produce a stable mesh carrier. Second, it fixes the mesh and UVs, then jointly optimizes UV-space PBR maps and environment lighting using differentiable rendering. The framework incorporates secondary-ray tracing to model one-bounce diffuse indirect illumination, enhancing relighting realism. This approach effectively decouples geometry, material, and lighting, enabling direct export of editable assets compatible with standard DCC workflows.
Key Results
- On DTU and Stanford-ORB datasets, the method achieves geometric errors below 0.3mm and 0.25mm respectively, outperforming many implicit surface methods. In relighting tests on Synthetic4Relight and Stanford-ORB, the average relighting error remains under 5%, demonstrating strong generalization. The exported assets are directly usable in typical content creation pipelines, supporting editing and scene composition.
- By fixing the mesh-UV carrier before material-light optimization, the model reduces ambiguity and improves material decomposition stability. Incorporating secondary-ray tracing captures local color bleeding and shadowing, resulting in more realistic relighting effects. Ablation studies confirm the importance of topology adaptation and fixed UV maps for detail preservation and stability.
- The approach maintains high geometric fidelity while enabling physically-based relighting, with robust performance across complex geometries and lighting conditions. It surpasses prior methods in producing explicit, editable assets suitable for downstream applications.
Significance
This work advances the state-of-the-art in multi-view 3D reconstruction by integrating explicit mesh refinement with physically-based material and lighting decomposition. It addresses longstanding challenges in achieving stable, relightable assets that are directly usable in production pipelines. The topology-adaptive framework and UV-PBR separation provide a practical solution for generating high-quality, editable 3D assets from images, facilitating applications in virtual reality, gaming, and digital twins. The ability to model indirect illumination without additional learned residuals marks a significant step toward realistic scene synthesis.
Technical Contribution
The paper introduces a novel topology-adaptive explicit mesh reconstruction method that dynamically refines geometry while maintaining UV consistency. It formulates a two-stage optimization: first, geometry and topology are optimized jointly; second, UV maps and PBR materials are refined independently, reducing mutual interference. The integration of secondary-ray tracing for diffuse indirect lighting within a fixed mesh and UV framework enables realistic relighting without complex residual networks. These innovations collectively enhance the stability, accuracy, and usability of reconstructed assets.
Novelty
This is the first work to combine topology-adaptive mesh refinement with explicit UV-PBR material decomposition in a staged framework. Unlike prior approaches that optimize neural fields or point primitives, this method maintains a stable, explicit mesh with consistent UVs, enabling direct asset export. The incorporation of secondary-ray tracing for indirect illumination within this explicit framework further distinguishes it from existing neural or hybrid methods. These contributions collectively push the boundary of multi-view asset reconstruction toward production-ready, relightable 3D models.
Limitations
- The method relies on multi-view images with known camera parameters, limiting its applicability to static scenes. Handling highly complex or extremely detailed geometries may introduce instability during topology updates. Computational costs remain significant, especially during the topology refinement stage, hindering real-time applications. Extending to dynamic scenes and single-view scenarios remains challenging and requires further research.
Future Work
Future directions include developing end-to-end deep learning models for automatic topology adaptation, reducing computational overhead, and enabling real-time reconstruction. Extending the framework to dynamic scenes with temporal coherence and handling single-view inputs are promising avenues. Additionally, integrating learned priors for better initializations and exploring multi-scale approaches could further improve detail preservation and efficiency.
AI Executive Summary
Reconstructing detailed, relightable 3D assets from multi-view images remains a core challenge in computer graphics and vision. Existing approaches often produce meshes that lack explicit material and lighting separation, limiting their utility in realistic rendering and editing. Neural implicit representations, while effective for surface reconstruction, struggle with direct asset extraction and physical interpretability. To address these issues, this paper introduces ExMesh++, a novel framework that combines topology-adaptive mesh refinement with explicit UV-PBR decomposition.
The first stage of ExMesh++ employs a dynamic mesh optimization process, where vertices are split or merged based on local geometric cues, ensuring detailed surface reconstruction while maintaining UV consistency. This results in a stable, high-quality mesh carrier. In the second stage, the mesh and UVs are fixed, and the system jointly optimizes physically-based material maps and environment lighting within the UV space, using differentiable rendering. A key innovation is the incorporation of secondary-ray tracing, which models one-bounce diffuse indirect illumination, capturing local shading effects realistically.
Experimental evaluations demonstrate that ExMesh++ achieves geometric accuracy surpassing 0.3mm on benchmark datasets, with relighting errors below 5%. The generated assets are directly compatible with standard digital content creation workflows, supporting editing, relighting, and scene composition. The approach effectively disentangles geometry, material, and illumination, enabling physically plausible and editable 3D assets from images.
This work significantly advances the field by providing a practical, end-to-end pipeline for high-fidelity, relightable asset reconstruction. Its core innovations—topology-adaptive mesh refinement, explicit UV-PBR optimization, and integrated indirect lighting—set new standards for multi-view 3D modeling, with broad implications for industry and research in virtual reality, gaming, and digital twins.
Deep Dive
Key Concepts
Topology-Adaptive Mesh
A mesh that dynamically adjusts its connectivity through vertex splitting and merging, enabling detailed surface refinement while maintaining UV consistency.
UV-PBR Maps
Physically-based rendering maps (albedo, normal, roughness, metallic) mapped onto UV coordinates, allowing realistic material representation and relighting.
Differentiable Renderer
A rendering process that computes gradients with respect to scene parameters, enabling end-to-end optimization of geometry and appearance.
Secondary-Ray Tracing
A technique to simulate indirect light by tracing secondary rays, capturing effects like color bleeding and soft shadows.
Explicit Mesh Representation
A 3D model stored as vertices and faces, directly manipulable and suitable for production workflows.
Open Questions Unanswered questions from this research
- 1 Current methods assume static scenes with known camera parameters; extending to dynamic or monocular scenarios remains unresolved. Handling highly complex geometries during topology updates can cause instability. Achieving real-time performance for large scenes is still a challenge, requiring further algorithmic optimization.
Applications
Immediate Applications
Virtual Asset Creation
Artists can generate high-quality, relightable 3D models from multi-view images for use in games, VR, and AR, streamlining content pipeline.
Scene Reconstruction
Reconstruct detailed scenes with physically accurate materials and lighting, enabling realistic virtual environment editing and simulation.
Long-term Vision
Automated Content Generation
Integrate deep learning for fully automated, end-to-end asset reconstruction from minimal inputs, transforming digital content workflows.
Abstract
Multi-view reconstruction extends beyond surface recovery to editable and relightable mesh assets. Such assets require well-formed topology, valid UV parameterization, and explicit PBR material maps. Existing surface reconstruction approaches optimize implicit fields, Gaussian primitives, or other intermediate representations. Converting them into such assets often requires surface extraction and texture baking. Inverse-rendering methods estimate materials and illumination, yet these components often remain tied to neural fields or point-based primitives rather than the final mesh. Joint optimization of geometry, materials, and lighting may also allow these variables to compensate for one another, leading to ambiguous decomposition. To address these limitations, we present ExMesh++, a staged framework for reconstructing relightable UV-PBR mesh assets from multi-view images. The first stage refines explicit mesh geometry and topology through adaptive vertex splitting and merging, while maintaining UV consistency as the topology changes. The second stage fixes the resulting mesh-UV carrier and optimizes UV-space PBR maps together with environment lighting. Building on this stable carrier, ExMesh++ models one-bounce diffuse indirect illumination through secondary-ray tracing with shared UV-PBR materials. Experiments demonstrate competitive geometry accuracy, strong relighting performance, and direct usability of the exported assets in standard DCC workflows.