DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising

3D object generation is a crucial research direction in computer vision and graphics, involving tasks that generate three-dimensional shapes from text descriptions. Traditional 3D generation methods mainly rely on geometric information, overlooking the semantic and functional structures of objects, leading to deficiencies in geometric fidelity and text alignment. In recent years, with the development of deep learning technology, neural network-based generation methods have gradually become mainstream, such as DreamFusion and ProlificDreamer. However, these methods typically focus only on generating whole objects without considering the relationships and semantic consistency between parts. To overcome these challenges, researchers have begun exploring part-level generation methods, introducing part decomposition and semantically grounded generation frameworks to improve the accuracy and consistency of generation. DreamPartGen was proposed in this context, achieving semantically grounded part-level 3D generation through collaborative latent denoising, providing new ideas for addressing the shortcomings of existing methods.

Existing text-to-3D generation methods often overlook the semantic and functional structures of objects when handling complex objects, leading to deficiencies in geometric fidelity and text alignment. Specifically, these methods typically focus only on generating whole objects without considering the relationships and semantic consistency between parts. Additionally, existing methods may produce unstable and inconsistent generation results when dealing with ambiguous or unclear text inputs. Achieving semantically consistent part-level generation while maintaining geometric fidelity is a significant challenge in current research.

The core innovation of DreamPartGen lies in its collaborative latent denoising framework, which unifies geometric, visual, and relational reasoning through the introduction of Duplex Part Latents (DPLs) and Relational Semantic Latents (RSLs). Specifically, DPLs are used to jointly model the geometry and appearance of each part, while RSLs capture inter-part dependencies derived from language. Through a synchronized co-denoising process, DreamPartGen ensures geometric and semantic consistency, enabling coherent, interpretable, and text-aligned 3D synthesis. Compared to existing methods, DreamPartGen achieves significant improvements in geometric fidelity and text alignment, offering new theoretical guarantees and engineering possibilities.

The methodology of DreamPartGen can be divided into several key steps:

• �� Introduction of Duplex Part Latents (DPLs): DPLs are used to jointly model the geometry and appearance of each part, capturing local geometric and visual details through 3D and 2D latent sequences.

• �� Introduction of Relational Semantic Latents (RSLs): RSLs capture inter-part dependencies derived from language, providing control signals for part interactions through global relational and local semantic tokens.

• �� Collaborative denoising process: Through a synchronized co-denoising process, DPLs and RSLs co-evolve under part-level and object-level synchronization, ensuring geometric and semantic consistency.

• �� Use of the large-scale PartRel3D dataset: The PartRel3D dataset provides rich functional and spatial relational triplets for explicit language-based supervision of inter-part relations.

In the experimental design, researchers used multiple benchmark datasets, including Objaverse, ShapeNet, ABO, and PartRel3D, to evaluate the performance of DreamPartGen. The baseline methods used in the experiments include Trellis, CLAY, HoloPart, and PartCrafter, which represent the latest advancements in the field of 3D generation. To assess the quality of the generated results, researchers adopted various metrics, including Chamfer Distance (CD), Earth Mover’s Distance (EMD), Render-FID, and Render-KID. Additionally, ablation studies were conducted to analyze the contribution of different components to the generation results.

Experimental results show that DreamPartGen performs exceptionally well across multiple benchmarks, significantly improving geometric fidelity by reducing Chamfer Distance by 53% and enhancing text-shape alignment by 20%. On the PartRel3D dataset, DreamPartGen surpasses previous baselines in geometric precision (53% reduction in CD, 33% reduction in EMD) and text-shape alignment (20% improvement in CLIP/ULIP). Additionally, in generalization tests for rare parts and unseen relation predicates, DreamPartGen outperforms prior baselines, improving Render-FID by 14.7-16.3%, CD by 68.2-71.2%, and ULIP-T by 39.6-47.9%. These results demonstrate DreamPartGen's robust capabilities in complex 3D structures.

Application scenarios for DreamPartGen include fine-grained part editing, articulated object generation, and mini-scene synthesis. Through its semantically grounded part-level generation framework, DreamPartGen provides fine control capabilities for these applications. Additionally, DreamPartGen can be applied to fields such as virtual reality and augmented reality, offering new solutions for 3D generation tasks in these areas. Its potential impact in academia and industry could be extensive and far-reaching.

Despite the significant advancements achieved by DreamPartGen in the field of 3D generation, there are still some limitations. First, DreamPartGen may encounter performance bottlenecks when handling very complex scenes, as the model's complexity and computational cost increase significantly. Second, the method's reliance on language descriptions may lead to inconsistent generation results when dealing with ambiguous or unclear text inputs. Additionally, there may still be issues with unstable generation or missing details in certain specific 3D shapes or structures. Future research directions include optimizing computational efficiency and improving the consistency and stability of generation results.