Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
Proposes Michelangelo framework utilizing shape-image-text aligned latent space with diffusion models, achieving superior quality and diversity in cross-modal 3D shape generation.
Key Findings
Methodology
This paper introduces a two-stage cross-modal 3D shape generation framework comprising the Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and the conditional Aligned Shape Latent Diffusion Model (ASLDM). SITA-VAE employs a perceiver-based transformer architecture to encode 3D shapes into a shared latent space aligned with CLIP-derived image and text features, utilizing contrastive learning to enforce semantic consistency. The decoder reconstructs detailed neural fields (occupancy or SDF) from these embeddings. ASLDM operates within this aligned latent space, learning a probabilistic mapping from image or text conditions to shape embeddings via a UNet-like transformer with diffusion mechanisms, enabling high-fidelity, diverse generation. The approach effectively bridges the domain gap between 2D/3D modalities, leveraging CLIP features to improve semantic alignment and detail preservation.
Key Results
- On the ShapeNet dataset and a newly collected 3D Cartoon Monster dataset, our model achieves an average IoU of 0.966, outperforming baselines such as 3DILG (0.950) and 3DS2V (0.955). In conditional generation, SI-S and ST-S scores reach 15.206 and 13.128 respectively, indicating enhanced semantic consistency and detail fidelity. The model demonstrates superior diversity and quality, validated through metrics like P-FID and P-IS, with ablation studies confirming the importance of the alignment space and diffusion mechanisms.
- Quantitative evaluations show that the proposed framework significantly improves shape reconstruction and conditional generation performance across multiple categories. The model maintains high semantic fidelity, generating shapes that closely match input conditions, with detailed textures and complex geometries. Visual comparisons reveal smoother surfaces and richer details compared to baseline methods, especially under text-conditioned scenarios.
- Ablation experiments demonstrate that the alignment in the latent space and the diffusion-based sampling process are critical for achieving high diversity and semantic accuracy. The use of CLIP features for cross-modal alignment effectively reduces the distribution gap, leading to more realistic and semantically aligned outputs. The model also exhibits robust generalization across different object categories and complex shapes.
Significance
This work addresses longstanding challenges in cross-modal 3D shape generation, notably the semantic gap between 2D/3D modalities and the difficulty in maintaining detail and diversity. By integrating a shared aligned latent space with advanced generative diffusion models, the framework achieves unprecedented levels of quality, diversity, and semantic fidelity. Its ability to generate detailed, semantically consistent 3D shapes from simple inputs like images or texts has profound implications for virtual content creation, industrial design, and digital arts. The methodology paves the way for more intelligent, flexible, and scalable 3D content generation systems, bridging the gap between human intent and machine synthesis.
Technical Contribution
The core technical innovations include: • Designing a shape-image-text aligned variational auto-encoder based on perceiver transformers, which leverages contrastive learning with CLIP features to embed 3D shapes into a shared semantic space; • Developing a conditional latent diffusion model built upon a UNet-like transformer architecture, enabling efficient probabilistic sampling within the aligned latent space; • Introducing a novel cross-modal alignment strategy that effectively reduces the distribution gap between 2D/3D modalities, facilitating high-fidelity, diverse shape generation; • Combining neural implicit representations with diffusion-based sampling, which allows detailed shape reconstruction and semantic consistency simultaneously. These contributions push the frontier of multimodal 3D generative modeling.
Novelty
This research is pioneering in integrating a shape-image-text aligned latent space with diffusion-based generative models for 3D shape synthesis. Unlike prior works that either focus solely on geometry or rely on post-hoc alignment, this framework explicitly models the semantic alignment across modalities during the encoding process. The use of CLIP features for multimodal embedding, combined with a diffusion model operating in this shared space, is a novel approach that significantly enhances both the quality and diversity of generated shapes. This is the first comprehensive attempt to unify these advanced techniques into a single, end-to-end framework for cross-modal 3D shape generation.
Limitations
- Despite the impressive results, the model's performance can degrade in highly complex or extremely detailed scenes, where subtle geometric nuances may be smoothed out or lost. This indicates a need for further refinement in the neural implicit decoder and the latent diffusion process.
- The training process requires large-scale, high-quality multimodal datasets with paired images, texts, and 3D models. Data scarcity or bias could limit the model's generalization to unseen categories or novel scenarios.
- Although inference is faster than optimization-based methods, the current implementation still involves substantial computational costs, especially for high-resolution shapes or real-time applications. Future work should focus on model compression and acceleration techniques.
Future Work
Future directions include enhancing the model's ability to handle more complex, fine-grained details, possibly through multi-scale or hierarchical representations. Exploring self-supervised or weakly supervised learning methods could reduce dependence on large annotated datasets. Extending the framework to incorporate additional modalities such as audio or tactile data may further enrich the generated content. Additionally, optimizing the inference pipeline for real-time applications and deploying the system on edge devices will be crucial for practical adoption.
AI Executive Summary
The generation of high-quality, diverse 3D shapes conditioned on visual or textual inputs remains a central challenge in computer vision and graphics. Traditional methods often struggle with maintaining semantic consistency and capturing intricate details, especially when translating from 2D images or natural language descriptions to 3D models. Existing approaches like GANs, VAEs, and diffusion models have made strides in single-modality scenarios but face significant hurdles in cross-modal settings due to the inherent distribution gap between different data types.
In response to this, the authors propose Michelangelo, a novel framework that fundamentally rethinks cross-modal 3D shape generation by introducing a shape-image-text aligned latent space. This shared space is designed to bridge the modality gap, enabling the model to understand and synthesize shapes that are both semantically accurate and geometrically detailed. The framework comprises two main components: the SITA-VAE, which encodes 3D shapes into an aligned latent space using a perceiver-based transformer architecture, and the ASLDM, a diffusion-based generative model that operates within this space to produce diverse, high-fidelity shapes conditioned on images or texts.
The core innovation lies in leveraging contrastive learning with CLIP features to align the semantic representations across modalities, ensuring that the generated shapes faithfully reflect the input conditions. The neural implicit decoder reconstructs detailed neural fields from these embeddings, capturing complex geometries and textures. Extensive experiments on ShapeNet and a newly collected 3D Cartoon Monster dataset demonstrate the effectiveness of the approach, with quantitative metrics such as an average IoU of 0.966 and significant improvements in shape fidelity and diversity over baseline methods.
This work has profound implications for virtual content creation, industrial design, and digital arts, providing a scalable, flexible, and semantically consistent solution for 3D shape synthesis. It addresses key limitations of prior models by integrating advanced multimodal alignment and diffusion techniques, opening new avenues for research and application in AI-driven 3D content generation. Future work will focus on enhancing model efficiency, expanding multimodal capabilities, and tackling more complex, real-world scenarios to further realize the potential of this innovative framework.
Deep Analysis
Background
随着深度学习在计算机视觉和图形学中的快速发展，3D形状的自动生成成为研究热点。早期方法多依赖手工特征或规则，难以应对复杂场景的多样性。近年来，神经场（Neural Fields）如Occupancy Networks、Signed Distance Functions（SDF）和Radiance Fields的出现，为高保真、拓扑无关的3D表示提供了新的技术路径。代表性工作包括Neural Radiance Fields (NeRF) 和DeepSDF，极大推动了3D重建和生成的发展。
与此同时，条件生成模型如GAN、VAE和扩散模型在图像和视频生成中取得巨大成功，但在3D生成中仍面临模态间分布差异和语义对齐的难题。尤其是在从2D图像或文本直接生成3D模型时，因信息维度差异大，导致生成结果常常偏离条件需求。近年来，结合对比学习和CLIP等多模态预训练模型，试图实现跨模态的语义对齐，成为研究热点。尽管如此，如何在保持几何细节的同时实现多模态信息的深度融合，仍是未解难题。
Core Problem
核心问题在于跨模态3D形状生成中的模态间分布差异。图像和文本的特征空间与3D潜在空间存在巨大差异，直接映射导致语义不一致、细节缺失或多样性不足。传统方法多采用逐步优化或后处理，效率低且难以保证语义一致性。设计一个统一的潜在空间，既能表达丰富的几何细节，又能融合多模态语义信息，是当前亟待解决的难题。此外，现有模型在多模态对齐和生成质量方面仍有提升空间，尤其在复杂场景和细粒度描述下表现不足。
Innovation
本研究的创新点主要包括：• 提出Shape-Image-Text对齐潜在空间，有效缓解模态间分布差异，提升语义一致性；• 设计基于Perceiver架构的SITA-VAE，通过对比学习实现多模态特征对齐，并利用Transformer解码器重建高细节的神经场；• 引入ASLDM，结合扩散机制与UNet式Transformer架构，在潜在空间中学习条件概率分布，实现多样化高质量3D生成；• 利用CLIP特征空间，增强模态间的语义对齐能力，提升跨模态生成的准确性和细节丰富度。这些创新结合深度学习的最新技术，为3D生成提供了全新的解决方案。
Methodology
- �� 输入：图像、文本和3D形状数据，通过预训练的CLIP模型提取特征。
- �� 3D形状编码：采样点云，利用Perceiver架构的Transformer编码为潜在向量，并通过对比学习将其与图像、文本特征对齐。
- �� 形状重建：利用Transformer解码器，将潜在向量映射回神经场（Occupancy或SDF），实现高细节重建。
- �� 模态对齐：采用对比损失，确保3D形状与图像、文本在特征空间中高度一致。
- �� 潜在扩散：在对齐空间中训练UNet式扩散模型，学习从噪声到潜在向量的条件分布，支持多样化生成。
- �� 生成流程：给定条件（图像或文本），通过扩散模型采样潜在向量，再由解码器重建3D形状，最终输出高质量模型。
Experiments
采用ShapeNet作为主要数据集，涵盖55个类别，包含约50K模型。还新收集了811个卡通怪兽模型，用于复杂场景验证。训练过程中，利用多模态配对数据增强文本描述（如模板和标签）以及多角度渲染图像。评估指标包括IoU、SI-S、ST-S、P-FID和P-IS，全面衡量重建精度、生成质量和多样性。模型参数在8块Tesla V100 GPU上训练约5天，采用DDIM采样器，50步生成高质量3D模型，推理时间约10秒。对比基线包括OccNet、ConvOcc、3DILG和3DS2V，验证模型在重建和条件生成上的优越性。
Results
模型在IoU指标上达到了平均0.966，优于主要对比模型。条件生成方面，SI-S和ST-S指标分别提升至15.206和13.128，显示出更强的语义一致性和细节表现。多模态对齐策略显著改善了文本条件下的生成效果，模型在复杂场景和细节丰富的模型上表现出色。消融实验验证了潜在空间对齐和扩散机制的关键作用，表明该方法在多样性、质量和语义一致性方面均优于现有技术。
Applications
该技术可广泛应用于虚拟现实、游戏开发、工业设计、动画制作和建筑可视化等领域。用户只需提供图像或文本描述，即可快速生成符合语义的3D模型，极大提高内容创作效率。未来，结合硬件加速和多模态感知，将推动虚拟内容的自动化生产，满足个性化定制和高效设计的需求。
Limitations & Outlook
尽管取得了显著进展，但模型在极端复杂或细节丰富的场景中仍存在细节平滑或模糊的问题。训练依赖大量多模态配对数据，数据偏差可能影响泛化能力。此外，模型推理速度虽快，但在大规模场景下仍需优化以降低计算成本。未来需加强模型的鲁棒性和泛化能力，提升在实际应用中的表现。
Plain Language Accessible to non-experts
想象你在一家大型工厂里，工厂里有许多不同的机器，每台机器都能制造不同的东西。以前，我们只能用手工操作每台机器，费时又费力。而现在，科学家们发明了一种智能机器人，它可以根据你给的图片或描述，自动设计出一台新机器，制造出你想要的东西。这就像你告诉机器人你想要一个玩具车，它就能自己设计出一辆漂亮的玩具车，甚至还能变出不同的款式。这个机器人背后有一个超级聪明的“脑袋”，它能理解你说的话和看你的图片，然后用它的“想象力”制造出符合你要求的3D模型。这个“脑袋”其实是用一种叫做“对齐潜在空间”的技术，把你说的和图片的意思都放在一个共同的“理解空间”里。这样，机器人就能更准确地理解你的需求，制造出既漂亮又符合要求的模型。这个技术让我们未来可以用简单的描述或图片，快速得到复杂的三维模型，就像魔法一样方便。
ELI14 Explained like you're 14
想象你在玩一个超级厉害的3D打印机，它可以根据你说的话或者画的图，自动帮你做出各种模型。以前，要做一个复杂的模型，比如一只怪兽，可能要花很多时间和手工操作。而现在，这个新技术就像给打印机装上了“聪明的大脑”，它能理解你说的每一句话，或者你画的每一幅图，然后用它的“想象力”快速做出模型。这个“聪明的大脑”其实是用了一种特别的技术，把你说的和画的内容都放在一个“共同理解的空间”里，这样它就能更准确地知道你想要什么。比如你说“一个有大眼睛的怪兽”，它就能用这个技术，生成一个符合描述的3D怪兽模型。这个过程就像你告诉朋友你的想法，然后朋友用魔法帮你变出你想要的东西一样，非常神奇。未来，我们可以只用一句话或者一张图片，就能得到想要的3D模型，省时又方便，像魔法一样神奇！
Glossary
Neural Fields (神经场)
一种用神经网络表示3D形状的方法，通过隐式函数描述几何和光照信息，支持拓扑无关的高保真重建。
论文中用来表示3D形状的核心技术。
CLIP (Contrastive Language-Image Pretraining, 对比语言-图像预训练)
由OpenAI提出的多模态预训练模型，能将图像和文本映射到同一特征空间，实现跨模态语义对齐。
用于多模态特征提取和对齐。
Variational Auto-Encoder (VAE, 变分自编码器)
一种生成模型，通过学习潜在空间的分布，实现数据的压缩与重建，支持生成多样化样本。
用于编码和重建3D形状。
扩散模型 (Diffusion Model)
一种概率生成模型，通过逐步去噪实现高质量样本生成，近年来在图像和3D生成中表现突出。
用于潜在空间的高效采样。
对比学习 (Contrastive Learning)
通过最大化正样本对的相似度和负样本对的差异度，学习具有语义一致性的特征空间。
实现模态间的语义对齐。
潜在空间 (Latent Space)
模型内部的抽象特征空间，用于压缩和表达复杂数据的语义信息。
在模型中实现跨模态对齐和生成。
UNet (U-Net架构)
一种带有跳跃连接的卷积神经网络架构，擅长图像到图像的转换任务，也用于潜在空间的条件生成。
在扩散模型中用于潜在空间的学习。
神经隐函数 (Neural Implicit Function)
用神经网络表示的函数，用于描述几何边界和细节，支持高保真重建。
作为3D形状的表示形式之一。
多模态对齐 (Multimodal Alignment)
论文的核心技术之一。
ShapeNet (ShapeNet数据集)
一个大规模的3D模型数据库，涵盖多类别对象，用于训练和评估3D生成模型。
作为主要实验数据源。
IoU (Intersection over Union, 交并比)
衡量两个空间区域重叠程度的指标，值越大表示重叠越好。
用于评估3D重建的准确性。
P-FID (Point-FID)
基于点云的生成质量指标，衡量生成模型的逼真度。
用于评估生成3D形状的质量。
P-IS (Point Inception Score)
衡量生成样本多样性和质量的指标，值越高越好。
评估生成模型的多样性。
CLIP特征空间
由CLIP模型提取的图像和文本的共同特征空间，用于跨模态语义对齐。
实现模态间的语义一致性。
Open Questions Unanswered questions from this research
- 1 尽管引入了模态对齐潜在空间，但在极端复杂或细节丰富的场景中，模型仍可能出现细节平滑或模糊的问题。未来需要进一步提升模型的细节表达能力和鲁棒性。
- 2 目前模型训练依赖大量高质量的多模态配对数据，数据偏差或不足可能影响模型的泛化能力和语义一致性，如何减少对大规模配对数据的依赖仍是挑战。
- 3 模型推理速度虽优于优化式方法，但在大规模场景或实时应用中仍存在一定的计算成本，未来需优化模型结构以提升推理效率。
- 4 在多模态融合方面，如何更好地融合多源信息，提升跨模态理解和生成的深度，仍需深入研究。
- 5 模型在处理极端细节或特殊场景（如极小或极大尺度对象）时的表现尚需验证，未来应结合多尺度建模技术进行改进。
Applications
Immediate Applications
虚拟内容快速生成
设计师和开发者可以通过输入简单的图片或文本描述，快速生成符合需求的3D模型，用于游戏、动画和虚拟现实内容制作，显著提升效率。
工业设计与原型制作
工程师利用该技术根据设计草图或描述，快速生成高保真3D模型，加快产品开发流程，降低成本。
个性化定制与虚拟试衣
电商平台可根据用户描述或图片，自动生成个性化的3D商品模型，实现虚拟试穿和定制服务，提升用户体验。
Long-term Vision
智能内容创作平台
未来将发展出全自动化的虚拟内容生成系统，用户只需提供文字或图片，即可生成丰富多样的虚拟场景、角色和物品，推动数字内容产业变革。
虚拟现实与增强现实的普及
随着模型效率提升和多模态理解增强，虚拟环境中的对象生成将变得更加智能和个性化，实现沉浸式体验的普及。
Abstract
We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.
References (20)
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
J. Park, Peter R. Florence, Julian Straub et al.
ULIP: Learning Unified Representation of Language, Image and Point Cloud for 3D Understanding
Le Xue, Mingfei Gao, Chen Xing et al.
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, A. Blattmann, Dominik Lorenz et al.
ShapeNet: An Information-Rich 3D Model Repository
Angel X. Chang, T. Funkhouser, L. Guibas et al.
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models
Biao Zhang, Jiapeng Tang, M. Nießner et al.
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al.
LION: Latent Point Diffusion Models for 3D Shape Generation
Xiaohui Zeng, Arash Vahdat, Francis Williams et al.
3DILG: Irregular Latent Grids for 3D Generative Modeling
Biao Zhang, M. Nießner, Peter Wonka
Convolutional Occupancy Networks
Songyou Peng, M. Niemeyer, L. Mescheder et al.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion
Julian Chibane, Thiemo Alldieck, Gerard Pons-Moll
Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
Kevin Chen, C. Choy, M. Savva et al.
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo et al.
SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators
Andrew Luo, Tianqin Li, Wenhao Zhang et al.
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Aditya Sanghi, Hang Chu, J. Lambourne et al.
Perceiver: General Perception with Iterative Attention
Andrew Jaegle, Felix Gimeno, Andrew Brock et al.
Denoising Diffusion Implicit Models
Jiaming Song, Chenlin Meng, S. Ermon
CLOTH3D: Clothed 3D Humans
Hugo Bertiche Argila, M. Madadi, Sergio Escalera
Learning to Dress 3D People in Generative Clothing
Qianli Ma, Jinlong Yang, Anurag Ranjan et al.
Cited By (20)
A Photo to 3D Workflow for Generation of LOD3 Digital Building Representation
QuadLink: Autoregressive Quad-Dominant Mesh Generation via Point-Relation Learning
BrepForge: Factorized B-rep Synthesis via Wireframe Composition and Boundary-Conditioned Surface Instantiation
Beyond Voxel 3D Editing: Learning from 3D Masks and Self-Constructed Data
Repurposing 3D Generative Model for Autoregressive Layout Generation
HSRI: High-Fidelity Shape Representation with Image Guidance
From Visual Synthesis to Interactive Worlds: Toward Production-Ready 3D Asset Generation
MeshReGen: A Unified 3D Geometry Regeneration Framework
GroundUpV2: Sketch-based 3D modeling of multi-building scenes from prototyping to vertex-efficient representations
Animator-Centric Skeleton Generation on Objects with Fine-Grained Details
EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers
TOPOS: High-Fidelity and Efficient Industry-Grade 3D Head Generation
ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation
AssetGen: Deployable 3D Asset Generation at Interactive Speed
BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization
CubePart: An Open-Vocabulary Part-Controllable 3D Generator
NeuROK: Generative 4D Neural Object Kinematics
UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD
ISAP-3D: Identity-Slot Aligned Part-Aware 3D Generation
Grow3D: Hierarchical Next-Scale Octree Prediction for Fast and High-Fidelity 3D Shape Generation