Ditto: Building Digital Twins of Articulated Objects from Interaction
Ditto reconstructs articulated digital twins from before/after point clouds, reaching 0.72 whole-object Chamfer Distance on Shape2Motion.
Key Findings
Methodology
Ditto encodes pre- and post-interaction point clouds with PointNet++, fuses them using scaled dot-product attention, and decodes separate geometry and motion features. ConvONet-style voxel grids and feature planes condition implicit occupancy, segmentation, joint-type, and parameter decoders. Marching Cubes extracts part meshes, while dense predictions from mobile-part points are averaged to obtain a global joint model.
Key Results
- On the Synthetic Dataset, Ditto obtains 0.38 whole-object and 0.21 mobile-part Chamfer Distance; prismatic angle error is 0.06, while revolute angle and position errors are 0.72 and 0.03, outperforming Global Joint at 0.54, 37.7, 0.69, 52.0, and 0.13.
- On Shape2Motion, Ditto reaches 0.72 whole-object and 0.42 mobile-part Chamfer Distance, with prismatic error 0.08 and revolute angle/position errors 1.36/0.02. It outperforms Correspondence and Global Joint.
- Ablations show dense per-point articulation prediction is more accurate than Global Joint. Attention fusion outperforms Concat Fusion, while separate 3D geometry grids and 2D motion planes balance detail and efficiency.
Significance
The work turns articulated-object understanding into deployable digital-twin construction. Rather than producing only point labels or an implicit motion state, it jointly estimates segmentation, completed part geometry, joint type, and explicit articulation suitable for physics engines. This addresses a long-standing bottleneck in embodied AI: interactive assets are still commonly authored by artists and engineers. The same pipeline also offers a route from real-world observation to usable AR/VR objects.
Technical Contribution
Ditto combines ConvONet local implicit fields, PointNet++ two-stream encoding, and cross-observation attention. Geometry uses a 3D voxel grid and occupancy field; articulation uses 2D feature planes and dense local predictions. It represents prismatic joints with axis up and displacement cp, and revolute joints with axis ur, projection direction dr, distance hr, and angle cr. Training combines BCE, angular, L1 state, displacement, and rotation-matrix losses, enabling explicit URDF-style extraction.
Novelty
Unlike A-SDF, which is category-level and represents articulation implicitly, Ditto explicitly reconstructs each part and its kinematic relation in a category-agnostic setting. Unlike Shape2Motion-style point-cloud methods, it produces compact meshes usable for collision computation. Its key methodological novelty is dense local joint prediction followed by mobile-part voting, rather than unstable global regression.
Limitations
- The method assumes that only one part moves per interaction and models only 1D prismatic or revolute joints. It therefore does not directly cover simultaneous multi-joint motion, articulated chains, or deformable objects.
- Performance depends on partial point-cloud quality and cross-observation correspondence. Occlusion, depth noise, unseen geometry, and joint types outside the training formulation may cause incorrect meshes or axes.
- Evaluation uses four categories from each of two datasets; real-world demonstrations are mainly qualitative, and physical properties such as mass and friction are assigned default values rather than inferred.
Future Work
Future research should handle multiple moving parts, multi-degree-of-freedom chains, and long interaction sequences. RGB, tactile sensing, action information, uncertainty estimation, and learned physical parameters could reduce ambiguity and improve sim-to-real transfer. More efficient neural fields and larger real-world benchmarks are also needed for scalable asset acquisition.
AI Executive Summary
Robots and mixed-reality systems need more than a static picture of a cabinet or drawer: they need separate parts, collision geometry, and a correct account of how those parts move. Existing pipelines rely heavily on artists, while point-cloud methods often stop at segmentation and single images cannot disambiguate rotation from translation. Ditto treats interaction as supervision: it compares observations before and after a part is moved.
The system uses PointNet++ to encode both point clouds and scaled dot-product attention to connect them. ConvONet-style local implicit representations then predict occupancy, static/mobile segmentation, joint type, and joint parameters. Geometry is represented with a 3D feature grid, motion with 2D feature planes. Dense local joint predictions are aggregated over the mobile part; thresholding and Marching Cubes produce explicit meshes that can be placed in a physics engine.
Ditto achieves 0.38 whole-object Chamfer Distance on the Synthetic Dataset and 0.72 on Shape2Motion, with revolute position errors of 0.03 and 0.02 respectively. It outperforms Correspondence and Global Joint baselines and demonstrates real-object reconstruction for simulation. The main caveats are the one-moving-part assumption, partial observations, limited categories, and default physical properties. Nevertheless, the paper shows a compelling route from observing an action to automatically creating an interactive digital asset.
Deep Analysis
Background
Synthetic data supports vision and embodied AI, but reality gaps remain. SLAM and 3D reconstruction primarily recover static geometry, whereas robot simulators require articulated parts, collision meshes, and kinematic trees. A-SDF models articulated shape implicitly; Shape2Motion and related methods reason over point clouds. These advances leave open the problem of automatically producing a complete, explicit, category-agnostic asset.
Core Problem
Given partial point clouds P1 and P2 before and after interaction, the system must segment static and mobile parts, reconstruct their geometry, identify a prismatic or revolute joint, and estimate its axis, pivot, and relative state change. Occlusion, correspondence, and the ambiguity of single-view articulation make this difficult.
Innovation
- ��Two-stream PointNet++ encoding with attention-based cross-observation fusion.
- ��A 3D voxel grid for detailed geometry and 2D feature planes for motion reasoning.
- ��Dense local prediction of segmentation, joint type, and parameters, followed by voting.
- ��Explicit mesh and articulation extraction using Multiresolution IsoSurface and Marching Cubes, targeting physics engines rather than point-cloud visualization alone.
Methodology
- ��Input: partial point clouds before and after interaction.
- ��Encoding: PointNet++ yields f1 and f2; attention computes Attn12=softmax(f1f2^T/√d)f2, then concatenates features.
- ��Propagation: separate PointNet++ decoders produce geometry and articulation features.
- ��Structured representation: geometry features are pooled into a voxel grid; articulation features into three 2D planes, processed by 3D/2D U-Nets.
- ��Implicit prediction: local sampling predicts occupancy, binary segmentation, joint type, and [up,cp] or [ur,dr,hr,cr].
- ��Optimization: BCE, angular, L1 state, displacement, and rotation losses are combined.
- ��Extraction: thresholded fields yield part meshes; mobile-point averages yield the global joint.
Experiments
Experiments use Abbatematteo et al.’s Synthetic Dataset and Shape2Motion, selecting four categories from each. Random start/end states and fused multiview depth images create incomplete observations. Baselines include category-specific A-SDF, FCGF-based Correspondence with nonlinear least squares, and Global Joint. Ablations test Concat Fusion, Share Feature, and Share Decoder. Metrics are whole/mobile Chamfer Distance, prismatic angle error, and revolute angle and position errors.
Results
On the Synthetic Dataset, Ditto reports 0.38 whole-object Chamfer, 0.21 mobile-part Chamfer, prismatic error 0.06, and revolute angle/position errors 0.72/0.03. On Shape2Motion, the values are 0.72, 0.42, 0.08, and 1.36/0.02. It improves over Global Joint’s whole-object Chamfer of 0.90 and Correspondence’s 2.22 on Shape2Motion. Ablations confirm that dense local prediction substantially reduces joint errors.
Applications
The output can be represented in URDF-like form and imported into physics engines for robot manipulation, door and drawer interaction, planning, and synthetic-data generation. Real-object demonstrations suggest simulation-to-real interaction transfer. Deployment requires depth observations with sufficient coverage, a visible interaction-induced motion, and the current single-mobile-part assumption.
Limitations & Outlook
The formulation covers only one moving part and 1D revolute or prismatic joints; multi-joint chains, deformable objects, contact dynamics, and physical-property estimation remain outside scope. Attention and repeated implicit queries also incur computation. Category coverage is limited, and real-world evidence is primarily qualitative. Future systems should use action sequences, multimodal sensing, learned mass/friction, uncertainty estimates, and larger real benchmarks.
Plain Language Accessible to non-experts
Imagine a clever carpenter who must recreate a cabinet inside a computer. First, the carpenter sees the cabinet before a drawer is opened, then sees it again afterward. Parts that stayed still are probably the cabinet body; the part that changed position is probably the drawer. The direction of change reveals the rail along which it moves.
The carpenter does not merely copy the visible outline. It divides the object into pieces, asks at many locations whether solid material exists, and fills in hidden regions. It also lets many small observers guess where the movement axis is and how far the part moved, then trusts the combined vote rather than one guess.
The result is not just a picture. It is a virtual cabinet whose parts can move and collide. A robot can practice opening the drawer in simulation before trying the real one. This saves manual modeling time and makes simulated worlds more realistic, although hidden surfaces and complex simultaneous movements can still confuse the system.
ELI14 Explained like you're 14
Imagine finding a real cupboard and wanting to use it as a game object. The game needs to know whether its door rotates or its drawer slides. Ditto watches it twice: once before someone moves a part and once afterward. The difference tells it what moved, in which direction, and by roughly how much.
Instead of saving a blurry cloud of dots, the program separates the cupboard body from the door or drawer and rebuilds their shapes. It asks lots of tiny questions: “Is there solid object here?” “Does this point belong to the moving part?” and “Where is the movement axis?” Combining many answers makes the final guess more reliable.
The finished object behaves like a real game prop: it has shape, separate pieces, and a movable connection. A robot can practice opening it in a virtual world, then try the learned behavior in reality. On the Synthetic Dataset, Ditto’s whole-shape error was 0.38, showing that it reconstructs more than a rough visual imitation.
There are rules, though: it works best when only one part moves, and unseen areas may be guessed incorrectly. If future versions watch several actions, they could automatically build much richer robot worlds and games!
Glossary
Implicit Neural Representation
A neural network continuously represents a shape or property over 3D coordinates instead of storing a fixed mesh. Querying a coordinate returns values such as occupancy.
Ditto uses implicit fields for geometry, segmentation, and articulation.
PointNet++
A hierarchical neural architecture that extracts features from local neighborhoods in point clouds. It is robust to unordered point sets.
It encodes the pre- and post-interaction observations.
ConvONet
An implicit reconstruction framework that conditions neural fields on local convolutional feature grids. It supports detailed continuous geometry.
Ditto extends it with two-stream fusion and motion decoders.
Chamfer Distance
A point-set metric based on nearest-neighbor distances in both directions; lower values indicate closer geometry.
It evaluates whole and mobile-part reconstruction.
Prismatic/Revolute Joint
A prismatic joint permits translation along an axis; a revolute joint permits rotation around an axis.
Ditto estimates these two 1D joint types.
Marching Cubes
An algorithm that extracts a triangle surface from a sampled scalar field or occupancy volume.
It converts predicted occupancy fields into part meshes.
Open Questions Unanswered questions from this research
- 1 How to infer several simultaneously moving parts and their kinematic tree from sparse interactions remains unresolved. A robust temporal correspondence mechanism is needed.
- 2 Real depth noise, reflection, and severe occlusion can corrupt correspondence and hidden geometry. The paper does not provide a large quantitative real-world benchmark.
Applications
Immediate Applications
Robot simulation asset generation
Researchers can capture depth point clouds before and after opening a cabinet or drawer, run Ditto, and import the resulting part meshes and joints into URDF and a physics engine for manipulation-policy training.
AR/VR interaction prototyping
Developers can convert everyday furniture into movable virtual objects with less manual modeling. The prerequisites are a depth sensor, a clear single-part interaction, and a pipeline for exporting the reconstructed model.
Long-term Vision
Automatic interactive-world construction
With multi-joint and sequential-action extensions, Ditto-like systems could scan homes and automatically create large libraries of interactive assets for embodied AI, games, digital twins, and mixed reality.
Abstract
Digitizing physical objects into the virtual world has the potential to unlock new research and applications in embodied AI and mixed reality. This work focuses on recreating interactive digital twins of real-world articulated objects, which can be directly imported into virtual environments. We introduce Ditto to learn articulation model estimation and 3D geometry reconstruction of an articulated object through interactive perception. Given a pair of visual observations of an articulated object before and after interaction, Ditto reconstructs part-level geometry and estimates the articulation model of the object. We employ implicit neural representations for joint geometry and articulation modeling. Our experiments show that Ditto effectively builds digital twins of articulated objects in a category-agnostic way. We also apply Ditto to real-world objects and deploy the recreated digital twins in physical simulation. Code and additional results are available at https://ut-austin-rpl.github.io/Ditto