Wonder3D: Single Image to 3D using Cross-Domain Diffusion
Wonder3D employs cross-domain diffusion to efficiently generate detailed textured meshes from a single image.
Key Findings
Methodology
Wonder3D introduces a cross-domain diffusion framework that jointly generates multi-view normal maps and color images conditioned on a single input image, camera parameters, and text embeddings. The approach employs a domain switcher to alternate between normal and color domains within a pre-trained stable diffusion model, avoiding re-training. Multi-view cross-attention mechanisms facilitate information exchange across views and domains, ensuring geometric and visual consistency. A geometry-aware normal fusion algorithm then extracts high-quality surfaces from the multi-view 2D representations by optimizing a signed distance function (SDF) with normal supervision, regularization, and outlier suppression. This pipeline leverages the strong priors of large-scale pre-trained diffusion models, combined with geometric constraints, to produce detailed, consistent 3D meshes efficiently.
Key Results
- On the Google Scanned Object dataset, Wonder3D achieves a Chamfer distance of 0.0199 and a volume IoU of 0.6244, outperforming baseline methods like Zero123 and SyncDreamer. The entire reconstruction process takes only 2-3 minutes, representing a significant speedup.
- The model demonstrates robust generalization across diverse object categories and styles, producing high-fidelity textures and detailed geometries even from sparse views or stylized images.
- Ablation studies confirm that the cross-domain attention and geometry-aware normal fusion are critical for high-quality results, with each component contributing to the overall performance.
Significance
This work addresses the longstanding challenge of balancing reconstruction quality, efficiency, and consistency in single-view 3D modeling. By integrating pre-trained diffusion priors with geometric optimization, it offers a practical solution for rapid, detailed 3D content creation. The method reduces reliance on large 3D datasets, making high-fidelity reconstruction accessible for real-time applications in AR/VR, gaming, and robotics. Its ability to generate detailed meshes from minimal input paves the way for democratizing 3D content generation, fostering innovation in digital design and virtual experiences.
Technical Contribution
The key technical innovation lies in extending stable diffusion models to operate across domains via a domain switcher and cross-domain attention, enabling joint generation of normals and colors. The geometry-aware normal fusion algorithm introduces a robust surface extraction method that combines multi-view normal maps with SDF optimization, incorporating normal supervision and outlier suppression. This integrated pipeline achieves high detail, consistency, and speed, setting a new benchmark for single-view 3D reconstruction.
Novelty
This is the first work to unify multi-view normal and color image generation within a cross-domain diffusion framework, leveraging multi-view attention to ensure consistency. Unlike prior methods that rely solely on color images or per-shape optimization, Wonder3D combines generative priors with geometric constraints, enabling rapid, detailed, and consistent 3D reconstructions from a single image without extensive optimization or large 3D datasets.
Limitations
- The approach struggles with scenes involving severe occlusion or complex geometries, where generated normal maps may lack accuracy, leading to incomplete or distorted reconstructions.
- Performance degrades when input images are highly noisy or blurry, as the diffusion model's priors are less effective under such conditions.
- The method still requires multiple views for optimal results; single-view inputs with extreme angles may produce less accurate meshes.
Future Work
Future directions include integrating multi-modal cues such as depth or tactile data to improve robustness, extending the framework for real-time applications, and enhancing the normal fusion algorithm to better handle complex geometries and occlusions. Additionally, scaling the model to larger, more diverse datasets could further improve generalization and detail fidelity.
AI Executive Summary
Wonder3D represents a significant advance in single-view 3D reconstruction, leveraging a novel cross-domain diffusion framework to produce high-fidelity textured meshes efficiently. Traditional methods often face a trade-off between quality and speed, with optimization-based approaches being slow and direct inference models lacking detail. In contrast, Wonder3D employs a multi-view normal and color generation scheme conditioned on a single image, using a domain switcher to operate seamlessly across domains within a pre-trained stable diffusion model. Multi-view cross-attention mechanisms ensure the generated normal maps and color images are geometrically and visually consistent, addressing a key challenge in multi-view synthesis.
The core innovation lies in the geometry-aware normal fusion algorithm, which robustly extracts surfaces by optimizing a signed distance function guided by multi-view normal supervision, regularization, and outlier suppression. Extensive experiments on the Google Scanned Object dataset demonstrate that Wonder3D achieves a Chamfer distance of 0.0199 and a volume IoU of 0.6244, outperforming several baseline methods. Remarkably, the entire reconstruction process takes only 2-3 minutes, making it suitable for real-time applications.
This approach significantly impacts fields like virtual reality, robotics, and digital content creation by enabling rapid, detailed, and consistent 3D model generation from minimal input. It reduces reliance on large 3D datasets and extensive optimization, democratizing high-quality 3D content production. While current limitations include handling highly complex scenes and noisy inputs, future work aims to incorporate multi-modal data and improve robustness, pushing the boundaries of real-time, high-fidelity 3D reconstruction.
Deep Dive
Abstract
In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images.Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works directly produce 3D information via fast network inferences, but their results are often of low quality and lack geometric details. To holistically improve the quality, consistency, and efficiency of image-to-3D tasks, we propose a cross-domain diffusion model that generates multi-view normal maps and the corresponding color images. To ensure consistency, we employ a multi-view cross-domain attention mechanism that facilitates information exchange across views and modalities. Lastly, we introduce a geometry-aware normal fusion algorithm that extracts high-quality surfaces from the multi-view 2D representations. Our extensive evaluations demonstrate that our method achieves high-quality reconstruction results, robust generalization, and reasonably good efficiency compared to prior works.