PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent
PartMat employs a single global latent to achieve efficient, material-aware 3D part decomposition, surpassing existing methods in accuracy and scalability.
Key Findings
Methodology
This paper introduces PartVAE, combining variational autoencoder and Transformer architectures to learn a unified multi-part geometry representation. A conditional diffusion model generates parts conditioned on reference images and overall geometry, while reinforcement learning optimizes material boundary alignment and overlap suppression. A sparse voxel flow matching model refines geometric details, enabling boundary segmentation along material edges. The approach decouples inference complexity from part count, achieving high accuracy and efficiency in material-aware decomposition.
Key Results
- On a dataset of approximately 300,000 material-annotated furniture shapes, PartMat achieves a [email protected] score of 68.2%, outperforming baselines. Chamfer Distance is as low as 4.9×10^-4, with inference latency remaining nearly constant across 1 to 32 parts, demonstrating excellent scalability.
- Reinforcement learning-based alignment significantly improves boundary clarity and spatial consistency, reducing overlaps and errors. Geometric detail recovery via sparse flow matching outperforms traditional voxel methods, especially in microstructures and thin features.
- In image-guided material generation, PartMat surpasses competing methods in geometric fidelity and material correspondence, with the lowest CD and highest [email protected], supporting up to 32 material regions efficiently, validating its practical potential.
Significance
This work advances the paradigm from function-based to material boundary-guided 3D decomposition, enabling precise, editable, and scalable asset creation for interior design, virtual try-on, and industrial applications. By encoding multiple parts into a single latent, it reduces computational costs while maintaining high geometric and material fidelity. The integration of reinforcement learning and detail restoration techniques pushes forward the seamless fusion of material perception and geometric editing, opening new avenues for intelligent 3D content creation and manipulation.
Technical Contribution
The paper pioneers a global latent space for multi-part, material-aware 3D decomposition, integrating VAE, Transformer, diffusion, and reinforcement learning. It introduces a novel architecture that decouples part count from inference complexity, employs differentiable rewards for boundary optimization, and combines sparse flow models for high-fidelity geometry recovery. These innovations collectively enable fast, accurate, and editable 3D asset generation, surpassing prior state-of-the-art in efficiency and precision.
Novelty
The key novelty lies in representing multiple material parts within a single global latent, avoiding per-part decoding bottlenecks. The reinforcement learning-based boundary optimization and sparse flow refinement are new contributions that significantly improve segmentation accuracy and geometric detail. Unlike prior methods relying on geometric or semantic cues, this approach directly encodes material boundaries, enabling scalable, high-quality, editable 3D decomposition.
Limitations
- The model struggles with extremely thin or micro-structured geometries, where boundary details are challenging to capture accurately. Further high-resolution refinement is needed for such cases.
- Training requires large datasets with detailed material annotations, which are costly to acquire and limit generalization to unseen materials.
- In complex scenes with drastic geometric or material variations, the sparse flow refinement may produce artifacts, necessitating more robust algorithms.
Future Work
Future directions include integrating multi-modal cues such as physical properties and user feedback to enhance segmentation accuracy. Developing self-supervised learning strategies could reduce reliance on annotated data. Extending the framework to dynamic scenes and complex material interactions will broaden its applicability, pushing toward fully autonomous, high-fidelity 3D content generation.
AI Executive Summary
The generation and editing of 3D assets have become central challenges in computer graphics and vision. Traditional methods primarily focus on geometric or semantic segmentation, which often neglect the importance of material boundaries critical for realistic rendering and physical simulation. Existing approaches either segment based on shape features or generate parts independently, leading to inefficiencies and inaccuracies, especially when handling complex scenes with multiple materials.
To address these limitations, the authors propose PartMat, a novel framework that leverages a single global latent representation to perform material-aware 3D part decomposition efficiently. The core architecture combines a variational autoencoder (PartVAE) with Transformer-based encoders, enabling the encoding of multi-part geometry into a compact, unified latent. This latent is decoded in a single pass, decoupling inference complexity from the number of parts. A conditional diffusion model generates parts conditioned on reference images and overall geometry, while reinforcement learning refines material boundaries by optimizing spatial alignment and overlap suppression through differentiable rewards.
Furthermore, to recover high-frequency geometric details, a sparse voxel flow matching model with part attention is employed, ensuring micro-structures and thin features are preserved. Extensive experiments on a dataset of 300,000 furniture models demonstrate that PartMat outperforms existing baselines in material decomposition accuracy, with a [email protected] of 68.2%, and maintains consistent inference speed regardless of part count. The integration of RL-based boundary optimization and geometry refinement significantly enhances the quality of generated assets.
This work represents a significant step toward scalable, editable, and material-aware 3D asset generation, with broad implications for virtual design, manufacturing, and digital content creation. Despite its strengths, challenges remain in handling extremely detailed geometries and reducing data dependency, guiding future research toward more robust, self-supervised, and dynamic scene applications.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Part-level 3D generation has recently attracted increasing attention for producing structured and editable 3D assets. However, existing methods typically decompose objects according to functional semantics rather than the editable material boundaries (e.g., fabric, wood, metal) required in practical 3D applications such as interior design. Additionally, current methods often generate parts independently, causing computational costs to scale linearly with the part count. To address these limitations, we present PartMat, an efficient material-aware 3D part decomposition pipeline that represents multi-part geometry with a single global latent. Given a reference image and a single whole-object geometry, PartMat decomposes the object into parts that follow material boundaries. First, we propose PartVAE to learn such a unified representation and decode all material parts in a single forward pass, thereby decoupling inference cost from the number of parts. Second, with this representation, a diffusion model is trained for part generation and refined via reinforcement learning for accurate material assignment and overlap suppression. Finally, to recover fine-grained geometric details, we introduce a sparse-voxel flow-matching model with part attention for geometry post-processing. Extensive experiments demonstrate that PartMat significantly outperforms existing baselines in material-aware decomposition accuracy and achieves comparable geometric quality, while maintaining efficient inference.