SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge
SPARK combines VLMs, DiT, and differentiable kinematics for single-image sim-ready reconstruction, reaching 0.3959 Chamfer Distance.
Key Findings
Methodology
SPARK takes one RGB image and uses GPT-4o to infer URDF links, hierarchy, joint type, axis, origin, and limits; Gemini 2.5 Flash Image generates part references and an open-state image. DINOv2 features condition a Diffusion Transformer with local, global, and bidirectional parent-child attention to generate part meshes and a complete mesh. Differentiable forward kinematics, PyTorch3D rendering, silhouette losses, and regularization then refine continuous joint parameters.
Key Results
- On 50 images spanning 25 categories from GAPartNet, SPARK achieves CD 0.3959, [email protected] 0.4214, and [email protected] 0.8934. It surpasses PartCrafter at 0.4342/0.3600/0.8840, OmniPart at 0.4971/0.1928/0.8469, and URDFormer at 1.0556/0.0438/0.1762.
- For URDF estimation, SPARK obtains AxisErr 0.1577, PivotErr 0.1653, and TypeErr 0.0500, versus 0.5491/0.3529/0.2500 for Articulate-Anything. Thus axes, pivots, and joint types are substantially more accurate.
- The strongest gain at strict [email protected] indicates finer geometry. Open-state supervision and differentiable optimization improve motion consistency. The paper trains with open, closed, and half-open states from PartNet-Mobility, although the supplied text does not report complete ablation numbers.
Significance
SPARK addresses a central bottleneck in embodied AI and robotics: recovering not only an object's appearance, but also separable geometry, part hierarchy, and executable motion from one image. Conventional generators often produce fused meshes, while appearance-based segmentation ignores articulation and occluded geometry. By connecting VLM semantics, 3D generation, and kinematic refinement, SPARK moves reconstruction toward assets that can be loaded into URDF-based simulation and manipulation pipelines. This could reduce expert authoring costs and support scalable interactive asset creation.
Technical Contribution
The main contribution is converting VLM structural reasoning into explicit generative conditions. Local cross-attention preserves part-image correspondence; global attention maintains whole-object coherence; bidirectional parent-child attention propagates assembly information. Dual position embeddings preserve identity under part shuffling. Geometry is trained with Rectified Flow, using U*=Z0−Z1 as the target velocity. URDF refinement optimizes SO(3) motion through differentiable rendering, combining region overlap, edge alignment, and quadratic pose regularization.
Novelty
Unlike Articulate AnyMesh and DreamArt, which often generate or retrieve a whole shape before decomposition, SPARK directly synthesizes kinematic parts and the complete object from one image. Unlike structural generators requiring explicit kinematic parameters, it obtains an initial URDF through VLM reasoning. Its fundamental novelty is a closed loop linking language-level semantic structure, part-conditioned geometry generation, and differentiable motion supervision.
Limitations
- The pipeline depends on GPT-4o, Gemini, and generated reference images. Severe occlusion, unusual categories, symmetric views, or ambiguous motion semantics can corrupt the predicted hierarchy and open-state image, with errors then propagating into geometry and URDF.
- Training relies mainly on PartNet-Mobility, while evaluation uses only 50 GAPartNet images. Generalization to real photographs, non-rigid parts, complex multi-DOF mechanisms, and varied cameras remains insufficiently established.
- Training requires four NVIDIA H100 GPUs for roughly 60 hours and several large models, raising questions about inference cost, latency, reproducibility, and deployment on resource-constrained robotic systems.
Future Work
Future work should expand real-world, multi-view, multi-state, and multi-DOF data while reducing dependence on closed VLMs and synthetic reference images. Physics-aware collision, mass, friction, and dynamics losses could jointly refine meshes and URDFs. Uncertainty estimation, candidate structure search, lightweight models, and closed-loop evaluation on real robotic manipulation would further test reliability and practical value.
AI Executive Summary
Articulated objects such as drawers, doors, and laptops are essential for robots and interactive worlds, yet turning a photograph into a simulation-ready asset usually requires experts to separate parts, define parent-child links, and measure axes and pivots. Standard 3D generators favor fused meshes; template-based or multi-view systems are less suitable for a single image.
SPARK introduces a single-image pipeline. GPT-4o predicts a coarse URDF, while Gemini 2.5 Flash Image creates per-part references and an open-state image. A DINOv2-conditioned Diffusion Transformer then uses local, global, and parent-child hierarchical attention to generate both articulated parts and the complete mesh. Differentiable forward kinematics and PyTorch3D rendering compare the predicted motion with the open-state reference, refining joint origins and angles through region, edge, and regularization losses. Meshy supplies textures and ICP aligns them.
On 50 GAPartNet images across 25 categories, SPARK reaches CD 0.3959, [email protected] 0.4214, and [email protected] 0.8934, outperforming PartCrafter, OmniPart, and URDFormer. Its AxisErr, PivotErr, and TypeErr are 0.1577, 0.1653, and 0.0500, better than Articulate-Anything. The results suggest that semantic VLM priors plus kinematic constraints can turn visual resemblance into separable, movable, simulatable assets. Robustness to real images and computational cost remain open challenges.
Deep Analysis
Background
3D generation has evolved from meshes, NeRFs, and 3D Gaussian splats to high-quality models such as TripoSG, TRELLIS, and Hunyuan3D. Yet most produce fused objects, and post-hoc segmentation is unreliable under occlusion. PartCrafter and OmniPart generate parts but are not necessarily motion-aware. Articulate-Anything and URDFormer estimate URDF-like structures, but often rely on multiple states, templates, or explicit structural inputs.
Core Problem
Given one RGB image, the goal is to recover part meshes {Mk}, a complete mesh M, and URDF parameters: links, parent-child relations, joint types, axes, origins, and limits. The task is hard because a single view lacks depth and motion evidence; visual boundaries do not always equal kinematic boundaries; hidden geometry must be completed; and an incorrect hierarchy can simultaneously damage shape quality and simulation behavior.
Innovation
- �� VLM structural parsing: GPT-4o predicts semantic links and coarse URDF data, while Gemini generates part and open-state images.
- �� Multi-level DiT: DINOv2 features feed local, global, and bidirectional hierarchical attention for correspondence and consistency.
- �� Differentiable URDF refinement: forward kinematics and rendering fit the open-state silhouette using region, edge, and pose losses.
- �� Data curation: PartNet-Mobility over-segmented meshes are merged by URDF links, and open/closed/half-open states are sampled.
Methodology
- �� Parsing: from I0, predict K semantic parts, a structure graph, and an initial URDF.
- �� Conditioning: DINOv2 encodes global and per-part images; each part has N latent tokens.
- �� Generation: DiT uses Alocal=softmax(ZiZiᵀ/√C) and Aglobal=softmax(ZZᵀ/√C), plus parent-child attention.
- �� Flow matching: xt=(1−t)z0+tz1 with target velocity U*=Z0−Z1 and weighted squared error.
- �� Post-processing: Meshy generates textures and ICP aligns meshes.
- �� Motion refinement: optimize ξ=(Δt,Δθ) with silhouette region/gradient losses and λt||Δt||²+λθ||Δθ||².
Experiments
Training uses PartNet-Mobility, containing 2,347 objects in 46 categories. Non-watertight meshes are repaired, and open, closed, and half-open states are added. Training uses four NVIDIA H100 GPUs, batch size 48, learning rate 1×10^-4, and 1,000 epochs, taking about 60 hours. Shape baselines are PartCrafter, OmniPart, and URDFormer; URDF baselines are Articulate-Anything and Articulate AnyMesh. Evaluation uses 50 GAPartNet images from 25 categories and reports CD, [email protected]/@0.5, AxisErr, PivotErr, and TypeErr.
Results
SPARK leads on all reported shape metrics: CD 0.3959, [email protected] 0.4214, and [email protected] 0.8934. PartCrafter obtains 0.4342/0.3600/0.8840, and OmniPart 0.4971/0.1928/0.8469. For URDF estimation, SPARK reaches AxisErr 0.1577, PivotErr 0.1653, and TypeErr 0.0500, compared with 0.5491, 0.3529, and 0.2500 for Articulate-Anything.
Applications
The output contains separable meshes and URDF parameters, supporting robotic grasping, drawer pulling, door and laptop articulation, simulation-scene construction, and interaction modeling. Practical deployment requires reliable single-image parsing, acceptable VLM cost, and URDF-compatible simulators. Adding collision, mass, and friction parameters could extend the system to reinforcement learning and manipulation planning.
Limitations & Outlook
The method assumes that VLMs can correctly infer parts and motion semantics and that a generated open-state image adequately represents articulation. Occlusion, mirror ambiguity, rare mechanisms, non-rigid materials, and coupled joints may cause errors. The evaluation is small and does not fully cover real photographs or dynamic video. Dependence on GPT-4o, Gemini, Meshy, and H100-scale training also affects cost and reproducibility. Physics validation, confidence modeling, real-robot trials, and model compression are important next steps.
Plain Language Accessible to non-experts
Imagine SPARK as an intelligent furniture factory. A photograph arrives, and the first worker writes an assembly list: this is the cabinet, this is the drawer, this is the handle; the drawer belongs to the cabinet and should slide forward. That is the system's visual reasoning stage. Other workers then study both the full photograph and separate pictures of each component. They build an independent drawer and door while making sure colors, sizes, and connections still fit the whole object.
The factory then tests movement. It slides the drawer or rotates the door, takes a picture of the result, and compares it with a picture showing the furniture open. If the drawer moves sideways, the track is adjusted; if the door swings around the wrong place, its hinge is corrected. The final product is therefore not merely a picture-like model, but digital furniture that can open and close inside a computer simulation.
On 50 images from 25 categories, SPARK matched real shapes more closely than the compared systems. Its strict matching score was 0.4214, and its joint-type error was only 0.05. In short, it answers both “what does this object look like?” and “how should it move?”
ELI14 Explained like you're 14
Suppose you see a desk in a game and want to turn it into an interactive 3D object. A basic tool might make one solid desk-shaped block. It looks right, but the drawer cannot open and the cabinet door cannot rotate. SPARK acts like a clever game-building assistant: it studies the picture and lists the pieces, who is connected to whom, and which piece slides or spins.
Next, it makes small reference pictures for the pieces and generates their 3D shapes together. Why not create the whole desk at once? Because the drawer and door must exist separately so game code can control them. SPARK also keeps checking the full desk, so a drawer does not become too big or end up with the wrong color. The connection list works like building with LEGO: every piece stays in the right family.
Then comes the fun test. SPARK moves the door or drawer, turns that result into an image, and compares it with the expected open version. If the hinge is wrong, it changes the hinge; if the drawer travels too far, it pulls the motion back toward a sensible value. Tests show that its shapes and motion settings beat several other methods.
But it is not magic! A hidden part, a rare machine, or many linked movements can confuse it. What could help? More photos or videos, better physical checks, and real robot experiments. One day, a robot might look at a new cabinet and quickly learn how to open it—pretty cool, right?
Glossary
Vision-Language Model (VLM)
A model that understands images and language together. It converts visual evidence into semantic descriptions, relations, and action hypotheses.
GPT-4o predicts the URDF draft; Gemini 2.5 Flash Image generates part and open-state references.
Unified Robot Description Format (URDF)
A standard XML representation of links, joints, axes, origins, and motion limits. Simulators use it to interpret articulated structures.
SPARK predicts a coarse URDF and then refines its joint parameters.
Diffusion Transformer (DiT)
A Transformer architecture used to denoise or transport latent variables toward generated data. Here it maps conditioned latent tokens to 3D geometry.
The DiT jointly generates part-level and complete meshes.
Rectified Flow
A method that learns a continuous vector field transporting noise to data. SPARK uses xt=(1−t)z0+tz1 and target velocity U*=Z0−Z1.
It defines the training objective for geometry generation.
Differentiable Rendering
Rendering designed so image errors produce gradients with respect to scene or geometry parameters. It enables visual supervision of 3D motion.
PyTorch3D renders silhouettes used to optimize joint origins and angles.
Chamfer Distance
A nearest-neighbor distance between two 3D point sets; lower values indicate closer geometry. It measures geometric reconstruction fidelity.
SPARK reports 0.3959 on the GAPartNet evaluation.
Open Questions Unanswered questions from this research
- 1 A single image cannot uniquely reveal hidden depth, backside geometry, or the true articulation axis. Video, multiple views, or active observation may be required to reduce this ambiguity.
- 2 The supplied paper text does not fully quantify how VLM mistakes are detected or corrected, and complete ablation values are unavailable. Confidence estimation and candidate-graph search remain open.
- 3 Simulation also needs mass, friction, collision, and actuator parameters. Automatically recovering these physical properties from appearance is still largely unsolved.
Applications
Immediate Applications
Robotic manipulation asset creation
Researchers can provide an object image and obtain separable meshes plus a URDF draft for simulated door opening, drawer pulling, or lid manipulation. With calibrated collision, mass, and control parameters, SPARK can shorten environment-authoring time and accelerate training-data generation.
Interactive content production
Game, AR, and digital-twin teams can convert photographs into furniture or devices with independently movable components. Meshy supplies textures and ICP improves alignment, making the pipeline useful for rapid prototyping of interactive assets.
Long-term Vision
Automatic embodied-AI asset libraries
A future system could continuously process web images or robot-camera observations to build large libraries of articulated objects with geometry, hierarchy, and motion. Major obstacles include real-world generalization, parameter confidence, physics validation, and scalable quality control.
Abstract
Articulated 3D objects are critical for embodied AI, robotics, and interactive scene understanding, yet creating simulation-ready assets remains labor-intensive and requires expert modeling of part hierarchies and motion structures. We introduce SPARK, a framework for reconstructing physically consistent, kinematic part-level articulated objects from a single RGB image. Given an input image, we first leverage VLMs to extract coarse URDF parameters and generate part-level reference images. We then integrate the part-image guidance and the inferred structure graph into a generative diffusion transformer to synthesize consistent part and complete shapes of articulated objects. To further refine the URDF parameters, we incorporate differentiable forward kinematics and differentiable rendering to optimize joint types, axes, and origins under VLM-generated open-state supervision. Extensive experiments show that SPARK produces high-quality, simulation-ready articulated assets across diverse categories, enabling downstream applications such as robotic manipulation and interaction modeling. Project page: https://heyumeng.com/SPARK/index.html.