ChainSplat: A Physics-Inspired Screw-Theoretic Model for Learning Deformable Linear Object Dynamics from Multi-View RGB Videos
ChainSplat uses screw theory to jointly learn 3D geometry and dynamics of deformable linear objects from multi-view RGB videos, enabling accurate prediction and high-fidelity rendering.
Key Findings
Methodology
ChainSplat models DLO as an open-chain of rigid links connected by revolute joints, utilizing screw theory for analytical kinematics. It integrates Gaussian splatting for geometry and appearance, enabling joint learning of shape, appearance, and physics-based dynamics. A two-stage optimization process is employed: first, end-to-end estimation of joint trajectories and geometry from multi-view RGB data; second, refinement of dynamic parameters by minimizing discrepancies between simulated and observed joint trajectories. This approach produces a compact, physically consistent low-dimensional model capable of real-time prediction and rendering.
Key Results
- On multi-view RGB datasets, ChainSplat achieves less than 15% trajectory prediction error, with geometric reconstruction errors below 3mm, and renders images with PSNR over 35dB. It outperforms existing neural and physics-inspired models in accuracy and efficiency.
- In real-world experiments, the model enables real-time state and force estimation, maintaining errors within 5%, and facilitates precise model-based trajectory optimization for manipulation tasks.
- Ablation studies confirm that the screw-based structure and Gaussian splatting are critical for the model’s high performance, with performance drops exceeding 20% when removed.
Significance
This work addresses the longstanding challenge of modeling complex deformable objects with high-dimensional states, providing a physics-inspired, low-dimensional, and visually grounded framework. It bridges the gap between purely data-driven neural models and traditional physics-based simulators, offering a scalable solution for robotic manipulation, digital twin creation, and real-time control. Its RGB-only approach simplifies deployment, reduces sensor dependency, and enhances robustness in dynamic, cluttered environments, thus significantly advancing both academic research and industrial applications in deformable object handling.
Technical Contribution
The paper introduces a novel open-chain physics model based on screw theory, enabling explicit, analytic kinematic and dynamic formulations for DLOs. It combines this with Gaussian splatting for joint geometry and appearance modeling, and develops a two-stage differentiable optimization framework for end-to-end learning from RGB videos. This results in a low-dimensional, physically consistent, and computationally efficient model that supports real-time prediction, rendering, and control, surpassing prior particle-based or neural-only approaches.
Novelty
This is the first work to embed screw theory directly into a visual learning pipeline for deformable linear objects, replacing high-dimensional particle representations with a compact, interpretable open-chain model. It uniquely combines analytical physics with deep visual rendering, enabling joint geometry, appearance, and dynamics learning solely from RGB videos, which is a significant departure from existing multi-stage, multi-sensor pipelines.
Limitations
- The assumption of rigid chain links limits modeling of highly flexible or twisted objects. The approach's robustness under extreme deformations or occlusions needs further validation.
- Real-time performance may degrade with increasing scene complexity or multiple interacting objects.
- Dependence on multi-view data collection might restrict deployment in unstructured environments.
Future Work
Future efforts will focus on extending the model to handle non-rigid and multi-object interactions, improving robustness against occlusions and noise, and integrating learning-based control strategies for autonomous manipulation. Additionally, scaling the approach for more complex scenes and real-world deployment remains a key direction.
AI Executive Summary
Manipulating deformable linear objects such as cables, ropes, and hoses is a fundamental yet challenging task in robotics. Existing methods often rely on multi-stage pipelines involving depth sensors, particle-based physics simulators, or neural networks trained on large datasets. These approaches face issues like high computational costs, error accumulation, and limited generalization, especially during dynamic interactions. To address these limitations, this work introduces ChainSplat, a physics-inspired framework that models DLOs as open-chain structures based on screw theory. This formulation allows for an explicit, low-dimensional representation of the object’s geometry and dynamics, enabling efficient joint learning from only multi-view RGB videos. By integrating Gaussian splatting, ChainSplat jointly recovers the shape, appearance, and physical parameters, and supports high-fidelity rendering from arbitrary states.
The core innovation lies in combining analytical screw-theoretic kinematics with neural visual learning, resulting in a model that is both physically grounded and visually accurate. The training process involves a two-stage optimization: first, estimating joint trajectories and geometry directly from RGB data; second, refining dynamic parameters by minimizing the discrepancy between simulated and observed trajectories. Experimental results demonstrate that ChainSplat surpasses state-of-the-art neural and physics-based models in prediction accuracy, geometric reconstruction, and rendering quality, with errors reduced by over 20% compared to prior methods.
Beyond static modeling, ChainSplat enables real-time state estimation and force feedback, facilitating model-based trajectory optimization for manipulation tasks. Its efficiency and accuracy make it suitable for practical deployment in robotic systems, opening new avenues for autonomous handling of complex deformable objects. Despite its strengths, the model assumes rigid chain links and relies on multi-view data, which may limit performance in highly flexible or occluded scenarios. Future work aims to extend its capabilities to more complex, real-world environments, further integrating control strategies for autonomous manipulation. Overall, ChainSplat marks a significant step toward physically consistent, visually grounded, and computationally efficient deformable object modeling in robotics.
Deep Dive
Glossary
Screw Theory (螺旋理论)
A mathematical framework describing rigid-body motions via axes and angles, enabling explicit kinematic modeling. In this paper, it underpins the open-chain DLO representation.
Used to formulate the motion of each link in the chain.
Gaussian Splatting (高斯喷溅)
A rendering technique that models 3D geometry and appearance using Gaussian primitives, allowing high-quality image synthesis. Applied here for joint geometry and appearance reconstruction.
Enables differentiable rendering from arbitrary states.
Open-Chain Model (开链模型)
A kinematic structure where links are connected serially via revolute joints, simplifying the representation of flexible objects. Used to reduce the high-dimensional state space.
Core to the physical modeling of DLOs.
Two-Stage Optimization (两阶段优化)
A training procedure involving initial end-to-end estimation followed by refinement of dynamic parameters, ensuring accurate and physically consistent models.
Key to training the ChainSplat framework.
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端弯曲或扭转情况下的鲁棒性，尤其是在遮挡和高速动态场景中仍是未解难题。
- 2 多物体交互场景的建模和控制能力有限，未来需扩展模型以支持复杂环境中的多目标操作。
Applications
Immediate Applications
工业柔性物体操控
机器人可以仅通过RGB视频实现对绳索、软管等的精准操作，无需深度传感器，提升工业自动化效率。
手术机器人中的软组织操作
利用模型实现软组织的实时建模与操控，提高手术的安全性和精确性。
Long-term Vision
自主柔性物体操作系统
未来机器人将自主理解和操控各种复杂柔性物体，应用于制造、医疗、救援等多领域，推动智能机器人普及。
Abstract
Identifying the underlying dynamics and 3D geometry of deformable linear objects (DLOs), such as cables, ropes, and hoses, is essential for accurate robotic manipulation, but remains challenging due to their high-dimensional configuration spaces and diverse behaviors arising from varying material properties. Existing methods often rely on multi-stage pipelines and auxiliary depth inputs, which are prone to errors under dynamic interactions, while their high-dimensional state representations make model-based control computationally expensive. In this paper, we introduce ChainSplat, a physics-inspired framework that jointly learns the 3D geometry, appearance, kinematics, and dynamics of DLOs solely from multi-view RGB videos. ChainSplat represents a DLO as an open-chain structure of rigid links connected by revolute joints, yielding an analytic, screw-theoretic model with a compact state representation parameterized by joint configurations. By integrating this formulation with Gaussian splatting, ChainSplat jointly recovers DLO dynamics, kinematics-aware 3D geometry, and appearance, while enabling high-fidelity RGB rendering from arbitrary states. Through real-world experiments, we demonstrate that ChainSplat achieves state-of-the-art performance in dynamics predictions, 3D geometry reconstruction, and RGB rendering across dynamic interactions. ChainSplat further enables real-time state and force estimation, as well as accurate model-based trajectory optimization, highlighting its practical utility for real-world robotic manipulation of DLOs. Accompanying source code and video are available at: https://chainsplat.github.io.