CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
CLIP-Forge employs a two-stage training framework to generate 3D shapes from text without paired data, achieving high diversity and accuracy.
Key Findings
Methodology
CLIP-Forge utilizes a two-stage training process: first, training a shape autoencoder on unlabeled 3D data to embed shapes into a latent space; second, training a conditional normalizing flow model conditioned on CLIP image features extracted from shape renderings. During inference, text is encoded via CLIP text encoder into the same latent space, and the flow model samples shape embeddings conditioned on this text feature. The shape decoder then reconstructs 3D shapes. This approach leverages CLIP's cross-modal capabilities, avoiding paired datasets and expensive optimization at inference.
Key Results
- On ShapeNet, CLIP-Forge achieves a FID of approximately 2425.25, IOU of 0.7275, and classification accuracy of 83.33%, surpassing supervised baselines. It demonstrates robust zero-shot generalization, generating diverse shapes aligned with category and attribute descriptions.
- Qualitative results show the model can produce multiple diverse shapes from a single text prompt, interpolate smoothly between prompts, and capture semantic nuances. Quantitative metrics confirm high diversity and fidelity, with ablation studies validating the robustness of design choices.
- The model's ability to generate multiple outputs per prompt and control over attributes highlights its practical utility for creative applications, with potential for extension to different 3D representations like point clouds.
Significance
This work addresses the critical challenge of large-scale paired data scarcity in 3D shape generation by exploiting pre-trained image-text models. It enables efficient, flexible, and diverse shape synthesis directly from natural language, opening new avenues for AI-assisted design, gaming, and virtual content creation. Its fully feed-forward nature ensures fast inference, making it suitable for real-time applications. The approach also sets a foundation for future unsupervised and weakly supervised 3D generative models, bridging the gap between vision-language understanding and 3D content creation.
Technical Contribution
The paper introduces a novel framework combining a shape autoencoder trained on unlabeled data with a conditional normalizing flow conditioned on CLIP image features. This design allows direct text-to-shape generation without paired datasets. The method leverages CLIP's joint embedding space, rendering images as a bridge, and employs flow models like RealNVP with dimension-wise masking for efficient density estimation. The architecture supports multiple shape outputs per prompt and avoids inference-time optimization, representing a significant step forward in zero-shot 3D generation.
Novelty
This is the first approach to achieve zero-shot 3D shape generation solely based on pre-trained image-text models without requiring paired text-shape datasets. It innovatively bridges 3D shapes and natural language via rendered images and CLIP embeddings, using a probabilistic flow model for sampling. Unlike prior supervised or optimization-based methods, it offers a scalable, fast, and flexible solution that generalizes across categories and attributes, marking a new paradigm in 3D content synthesis.
Limitations
- Despite promising results, the generated shapes still lack fine details and textures, especially for complex geometries, necessitating future integration of local implicit methods.
- Dependence on CLIP's training data distribution limits performance on out-of-domain or rare concepts, requiring fine-tuning or multi-modal fusion for broader applicability.
- While inference is fast, the quality may degrade with highly detailed or textured shapes, and computational costs grow with model complexity. Further work is needed to enhance realism and detail fidelity.
Future Work
Future directions include integrating local implicit representations to improve detail and texture quality, extending the framework to incorporate multi-modal attributes like color and material, and exploring other 3D formats such as meshes. Additionally, fine-tuning CLIP on domain-specific datasets could enhance accuracy. Developing interactive tools for real-time editing and expanding the model's capacity for complex scenes are promising avenues to broaden practical applications in design, entertainment, and virtual reality.
AI Executive Summary
Generating 3D shapes from natural language has long been a challenging goal in AI, hindered by the scarcity of large-scale paired datasets. Traditional supervised methods require extensive labeled data, which is costly and limited in scope. Recent advances in image-text models like CLIP have demonstrated remarkable zero-shot capabilities in 2D image synthesis, inspiring researchers to explore their potential in 3D shape generation.
This paper introduces CLIP-Forge, a novel framework that leverages pre-trained CLIP models to bridge the gap between natural language and 3D geometry without requiring explicit text-shape pairs. The core idea involves a two-stage training process: first, training a shape autoencoder on unlabeled 3D data to learn a compact latent representation; second, training a conditional flow model conditioned on CLIP image features extracted from shape renderings. During inference, the text prompt is encoded into the same latent space via CLIP's text encoder, and the flow model samples shape embeddings conditioned on this text feature. These embeddings are decoded into 3D shapes, enabling rapid, diverse generation.
Experimental results on ShapeNet demonstrate that CLIP-Forge outperforms supervised baselines in metrics like FID, IOU, and classification accuracy, with FID around 2425.25 and accuracy exceeding 83%. Qualitative analyses show the model can generate multiple diverse shapes for a single prompt, interpolate smoothly between concepts, and accurately capture semantic attributes. Ablation studies confirm the robustness of design choices such as the autoencoder architecture and flow conditioning mechanisms.
This approach significantly advances the field by enabling scalable, flexible, and high-quality zero-shot 3D shape synthesis. Its efficiency and generalization potential open new opportunities in creative design, gaming, and virtual content creation, reducing reliance on costly labeled datasets. Future work aims to incorporate textures, improve detail fidelity, and extend to other 3D representations, promising a broad impact across AI-driven content generation.
Deep Dive
Abstract
Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape generation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive comparative evaluations to better understand its behavior.