ART: Articulated Reconstruction Transformer
ART employs a category-agnostic transformer to reconstruct complete 3D articulated objects from sparse multi-state RGB images, predicting geometry, texture, and articulation parameters.
Key Findings
Methodology
ART leverages a transformer-based architecture that encodes multi-view, multi-state RGB images into tokens, which are then mapped to learnable part slots via attention mechanisms. Each slot predicts a rigid part's geometry, texture, and explicit articulation parameters (motion type, axis, pivot). The model employs a joint decoding strategy to produce unified, interpretable 3D representations, trained on large-scale, diverse datasets with per-part supervision. The training incorporates SDF volume rendering to optimize geometry and appearance consistency, enabling physically meaningful outputs suitable for simulation. The approach effectively combines part-based prediction with global context sharing, ensuring robust generalization across categories and complex articulation structures.
Key Results
- On benchmarks like StorageFurniture and PartNet-Mobility, ART reduces Chamfer distance to 0.009, GIoU to 0.4717, and LPIPS to 0.049, outperforming state-of-the-art optimization and feed-forward baselines by over 20% in key metrics, demonstrating superior geometric and appearance accuracy.
- In sparse multi-view and multi-state scenarios, ART maintains high fidelity, with PSNR exceeding 27 dB and F-Score over 0.76, confirming its robustness and generalization ability across diverse object categories.
- The model's category-agnostic design enables it to handle unseen classes effectively, with qualitative results showing detailed, textured, and physically plausible reconstructions that are ready for simulation and downstream tasks.
Significance
This work addresses the longstanding challenge of fast, accurate, and category-independent 3D reconstruction of articulated objects from limited visual data. By integrating a transformer architecture with part-based modeling, it overcomes the limitations of traditional optimization-based methods—slow and fragile—and existing feed-forward models—category-restricted and less detailed. The resulting system significantly accelerates the creation of realistic digital replicas, facilitating applications in VR/AR, robotics, and digital twins. Its ability to generalize across categories and handle multi-state configurations marks a major step toward scalable, automated scene understanding and interaction, opening new avenues for research and industry deployment.
Technical Contribution
The core technical innovation lies in the design of a category-agnostic transformer that models articulated objects as assemblies of rigid parts, each with explicit geometric and articulation parameters. The introduction of learnable part slots and joint attention mechanisms enables the model to efficiently fuse multi-view, multi-state visual cues into a coherent, part-level 3D representation. The joint decoding of geometry, texture, and articulation parameters in a unified framework, combined with large-scale supervised training, results in a versatile model capable of generalizing across diverse object categories. Additionally, the use of SDF volume rendering for supervision ensures high-fidelity geometry and appearance, making the outputs directly usable for simulation and physical reasoning. This approach bridges the gap between static reconstruction and dynamic, articulated scene understanding, offering a new paradigm for 3D modeling.
Novelty
This research is the first to propose a category-agnostic, transformer-based framework for articulated object reconstruction directly from sparse multi-view images. Unlike prior works limited to specific categories or relying on slow optimization, ART employs a part-based prediction paradigm with learnable slots and joint attention, enabling holistic, detailed, and physically interpretable reconstructions. Its ability to predict explicit articulation parameters alongside geometry and texture, trained on large, diverse datasets, sets it apart as a significant advancement in the field.
Limitations
- The model's performance may degrade under extreme occlusion or when input views are too sparse, as limited visual cues hinder accurate part and motion inference.
- Training requires extensive annotated datasets, which are costly to produce, and the model's ability to generalize to unseen or highly complex articulation types remains to be fully validated.
- Current focus is on static articulation parameters; dynamic, time-dependent motion modeling and real-time inference are future challenges.
Future Work
Future directions include integrating temporal information for dynamic motion modeling, reducing reliance on large annotated datasets via self-supervised learning, and extending the framework to handle more complex articulation types and higher-resolution outputs. Additionally, improving robustness to occlusion and exploring real-time inference for interactive applications are key goals. These advancements will further broaden the applicability of ART in robotics, virtual environments, and industrial automation.
AI Executive Summary
The ART (Articulated Reconstruction Transformer) represents a significant leap forward in 3D modeling of articulated objects from limited visual data. Traditional approaches, relying heavily on optimization, are often slow and limited to specific categories, restricting their scalability. ART introduces a category-agnostic, end-to-end transformer framework that encodes sparse multi-view, multi-state RGB images into a set of learnable part slots. Each slot predicts a rigid part's geometry, texture, and explicit articulation parameters, such as motion type, axis, and pivot point. This part-based prediction enables the reconstruction of complex articulated objects with high fidelity, capturing both geometric details and kinematic structures. The model is trained on large-scale, diverse datasets with per-part supervision, leveraging volume rendering techniques to ensure geometric and appearance consistency. Quantitative results demonstrate that ART outperforms existing methods, reducing Chamfer distance to 0.009 and LPIPS to 0.049, while maintaining robustness across categories and sparse inputs. Its ability to generate physically interpretable, simulation-ready models opens new avenues for applications in virtual reality, robotics, and digital content creation. The approach's core innovation lies in combining the expressive power of transformers with a part-based, joint decoding strategy, enabling holistic and scalable articulated scene understanding. Future work aims to incorporate dynamic motion modeling, self-supervised learning, and real-time inference, further expanding its impact in both research and industry.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
We introduce ART, Articulated Reconstruction Transformer -- a category-agnostic, feed-forward model that reconstructs complete 3D articulated objects from only sparse, multi-state RGB images. Previous methods for articulated object reconstruction either rely on slow optimization with fragile cross-state correspondences or use feed-forward models limited to specific object categories. In contrast, ART treats articulated objects as assemblies of rigid parts, formulating reconstruction as part-based prediction. Our newly designed transformer architecture maps sparse image inputs to a set of learnable part slots, from which ART jointly decodes unified representations for individual parts, including their 3D geometry, texture, and explicit articulation parameters. The resulting reconstructions are physically interpretable and readily exportable for simulation. Trained on a large-scale, diverse dataset with per-part supervision, and evaluated across diverse benchmarks, ART achieves significant improvements over existing baselines and establishes a new state of the art for articulated object reconstruction from image inputs.