PWM-ArtGen: Part World Model for Articulated Object Generation

The evolution of 3D object reconstruction has transitioned from multi-view optimization to single-image generation, driven by deep learning advances. Early methods like PartNet-Mobility relied heavily on annotated datasets, which are costly and limited in diversity. Recent approaches, such as implicit surface representations and large-scale foundation models, have improved geometric fidelity but struggle with dynamic behaviors. The challenge remains to generate articulated objects that are both geometrically accurate and dynamically plausible, especially from a single image. Existing datasets like ACD and synthetic datasets like PartNet-Mobility provide valuable benchmarks but lack the scale and realism needed for robust generalization. Consequently, there is a pressing need for models that can leverage unannotated data effectively, capturing complex visual dynamics and kinematic structures in diverse real-world scenarios.

The core challenge in single-image articulated object generation is accurately inferring kinematic parameters and dynamic behaviors without explicit supervision. Static images lack explicit motion cues, making it difficult to determine joint types, axes, and ranges. Existing methods either depend on multi-view data or manual annotations, which are expensive and not scalable. The fundamental bottleneck is the scarcity of large-scale, high-fidelity, unannotated datasets that can support learning complex visual dynamics. This limits the ability of models to generalize to real-world objects with diverse structures and textures. Overcoming these issues requires innovative approaches that can learn from abundant unlabeled visual data while ensuring physically plausible and geometrically consistent outputs.

This work introduces several key innovations: 1) a diffusion-based joint model that learns the distribution of visual dynamics and kinematic parameters simultaneously, 2) independent diffusion timesteps enabling effective co-training on unlabeled data, 3) a visual dynamics regularizer (VDR) that enforces dynamic consistency, and 4) a large-scale high-fidelity dataset of 19.7k part-level image pairs. Unlike prior static or multi-view methods, PWM-ArtGen captures the interplay between motion and appearance, allowing for accurate, controllable generation of articulated objects from a single image. The integration of these components creates a scalable framework that bridges the gap between synthetic and real data, facilitating zero-shot generalization and dynamic realism.

• �� Input: single image and target part structure. • Part mask generator: uses SAM and GPT-4o to extract part masks and structural graphs. • Joint diffusion model: employs a diffusion transformer to predict kinematic parameters and visual dynamics, conditioned on static image features and part references. • Unsupervised training: utilizes independent diffusion steps for unlabeled data, with a coupled noise prediction network. • Visual regularization: applies VDR to align generated dynamics with real motion cues, enhancing physical plausibility. • Data: constructs 19.7k high-fidelity, unannotated part-level image pairs via rendering and editing, supporting large-scale training.

The model is trained on PartNet-Mobility and evaluated on ACD and unseen PartNet-Mobility objects. Metrics include dgIoU, dcDist, dCD, and articulation accuracy. Baselines include URDFormer, NAP, and SINGAPO. Ablation studies assess the impact of VDR and unlabeled data. Hyperparameters include 200k training steps, 12-layer diffusion transformer, and AdamW optimizer. Evaluation covers static and dynamic states, with zero-shot tests on out-of-distribution objects. Results show significant improvements over baselines, especially in complex scenarios, validating the effectiveness of the joint diffusion approach.

PWM-ArtGen achieves over 20% higher dgIoU and lower geometric errors compared to SOTA on both datasets. Zero-shot generalization maintains high accuracy, with 100% articulation graph correctness. Visualizations confirm realistic motion and geometry, even in occluded or textured objects. Ablation results demonstrate that removing VDR or unlabeled training reduces performance, highlighting their importance. The model also produces more coherent and physically plausible articulated behaviors, validated through quantitative metrics and qualitative assessments.

This approach can be directly applied to robotic manipulation, enabling robots to understand and generate complex articulated objects from minimal input. It also benefits virtual reality content creation, allowing rapid generation of realistic animated characters or objects. In industrial automation, it facilitates automatic modeling of machinery and components, reducing manual effort. Long-term, integrating multi-modal data like depth and temporal sequences could further enhance dynamic accuracy, supporting autonomous systems and digital twins in real-world environments.

Despite strong performance, the model faces challenges with highly occluded or extremely complex articulation patterns due to limited data diversity. Computational costs remain high because of the diffusion sampling process, restricting real-time deployment. Data collection and augmentation are expensive, and current datasets do not cover all possible object types or textures. Future work should focus on reducing inference latency, improving robustness in challenging scenarios, and expanding datasets to include more diverse and intricate articulations.