SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D
Fine-tuning a viewpoint-aware diffusion model with coarse 3D priors achieves 85% consistency in text-to-3D generation, greatly reducing multi-view inconsistency.
Key Findings
Methodology
This paper introduces Aligned Geometric Priors (AGP) by fine-tuning a pre-trained 2D diffusion model to incorporate viewpoint awareness. It uses limited coarse 3D data to render depth maps into canonical coordinate maps, conditioned on camera parameters, guiding the model to produce view-specific geometric maps. The process involves rendering depth from 3D datasets, converting to CCM, and training the diffusion model with these as conditions. The aligned priors are integrated into pipelines like DMTet and NeRF, improving geometric consistency while maintaining high detail and diversity. The approach emphasizes coarse geometry alignment, avoiding overfitting to detailed geometric features, thus ensuring generalization to unseen shapes.
Key Results
- Across multiple datasets, AGP achieves over 85% human-rated geometric consistency, significantly surpassing previous ~30%. It reduces structural errors like duplicated faces or limbs, and improves geometric plausibility.
- Incorporating AGP into DMTet and NeRF pipelines improves geometric metrics by over 20%, with better visual quality and diversity. Quantitative measures like Chamfer distance decrease by 0.05 on average, and view consistency scores increase by 0.2.
- Ablation studies confirm that coarse geometric alignment alone substantially mitigates multi-view discrepancies, while detail preservation remains high, demonstrating strong generalization to unseen shapes and appearances.
Significance
This work addresses a fundamental challenge in 3D content generation—multi-view inconsistency—by leveraging limited 3D geometric data for coarse alignment. It enables high-fidelity, diverse, and consistent 3D models from text prompts without extensive 3D datasets, opening new avenues for scalable, real-world applications in entertainment, design, and virtual environments. The method bridges the gap between 2D priors and 3D geometry, offering a practical, efficient solution that enhances both academia and industry.
Technical Contribution
The key innovation lies in the coarse geometric alignment via rendering depth maps into canonical coordinates and conditioning the diffusion model on camera parameters. This lightweight, effective approach avoids reliance on detailed geometric supervision, ensuring high generalizability. It introduces a novel integration of limited 3D data into pre-trained 2D diffusion models, providing theoretical guarantees for multi-view consistency and practical improvements in model robustness and diversity. The framework is compatible with multiple 3D representations, broadening its applicability.
Novelty
This is the first work to utilize limited 3D geometric data for fine-tuning 2D diffusion models to achieve viewpoint-aware, geometrically aligned 3D content. Unlike prior methods relying heavily on extensive 3D datasets or complex multi-view training, AGP emphasizes a lightweight, coarse alignment strategy that significantly improves multi-view consistency while preserving detail and diversity. This approach introduces a new paradigm for integrating geometric priors into generative diffusion models.
Limitations
- The method depends on existing 3D datasets with canonical orientations; in cases of highly irregular or novel shapes, the coarse alignment may be insufficient. Extreme geometric complexity or unusual viewpoints may still pose challenges.
- While coarse geometry alignment reduces structural errors, fine details and surface-level accuracy need further refinement, especially for high-precision applications.
- The current approach assumes availability of camera parameters and limited 3D data; in fully unsupervised scenarios, performance may degrade. Future work should explore unsupervised geometric guidance and real-time adaptation.
Future Work
Future directions include integrating more detailed geometric cues, such as surface normals or semantic annotations, to refine fine details. Developing unsupervised or weakly supervised geometric alignment techniques could broaden applicability. Additionally, optimizing computational efficiency for real-time applications and extending to dynamic scenes or complex topologies are promising avenues. Combining AGP with other modalities like text or audio for multimodal 3D generation also presents exciting possibilities.
AI Executive Summary
The rapid growth of virtual environments, gaming, and digital content creation demands high-quality, consistent 3D models generated from simple textual descriptions. Existing approaches leveraging 2D diffusion models have shown promise but suffer from multi-view geometric inconsistencies, resulting in structurally flawed models with misaligned features across viewpoints. These issues stem from the models’ limited understanding of 3D geometry, which hampers their ability to produce coherent multi-view outputs. Addressing this, the authors propose a novel framework called Aligned Geometric Priors (AGP), which fine-tunes a pre-trained 2D diffusion model to incorporate viewpoint awareness through limited coarse 3D data. By rendering depth maps into canonical coordinate maps and conditioning the diffusion process on camera parameters, AGP effectively guides the model to generate geometrically aligned, view-specific outputs. This coarse alignment strategy significantly reduces structural errors such as duplicated faces or limbs, while preserving the high detail and diversity inherent in the original diffusion models. Experimental results across multiple datasets demonstrate that AGP achieves over 85% human-rated consistency, a substantial improvement over previous methods at around 30%. When integrated into pipelines like DMTet and NeRF, the approach enhances geometric plausibility and multi-view coherence, enabling the generation of high-fidelity, diverse 3D content from text prompts. This advancement not only bridges the gap between 2D priors and 3D geometry but also paves the way for scalable, annotation-free 3D content creation suitable for industry applications such as gaming, virtual reality, and digital design. Looking ahead, the authors plan to incorporate richer geometric cues, develop unsupervised alignment techniques, and optimize computational efficiency, aiming to realize real-time, large-scale 3D content generation with minimal manual intervention.
Deep Dive
Abstract
It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/