CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects
CARTO achieves category- and joint-agnostic 3D reconstruction from a single stereo image, improving mAP 3D IOU50 by 20.4%.
Key Findings
Methodology
CARTO employs an implicit object-centric representation with a unified geometry and joint decoder capable of handling multiple categories and joint types. It integrates stereo RGB encoding, depth prediction, and interest point detection to extract per-pixel shape and joint codes in a single forward pass. The shape decoder, based on multi-layer perceptrons, predicts 3D shape, size, pose, and joint type, while the joint decoder regresses discrete joint types and continuous joint states. Regularization in shape and joint latent spaces enforces smoothness and generalization. The system is trained on simulated data and successfully transfers to real-world scenes, outperforming two-stage pipelines in accuracy and speed.
Key Results
- On unseen instances, the model improves mAP 3D IOU50 by 20.4%, with inference at 1Hz on a NVIDIA TITAN XP for scenes with up to eight objects. Compared to A-SDF, it reduces reconstruction error by ~15%, with joint type accuracy exceeding 90%. The model maintains robustness under occlusion and clutter, demonstrating strong generalization to novel objects and articulation states in both simulated and real datasets.
Significance
This work advances scene understanding by enabling rapid, category-agnostic, multi-object articulated reconstruction from a single image. It addresses the limitations of traditional multi-stage approaches, reducing complexity, inference time, and cumulative errors. The ability to generalize across categories and unseen instances opens new avenues for autonomous robots, AR/VR applications, and industrial automation, where real-time, flexible scene comprehension is critical.
Technical Contribution
The paper introduces a universal implicit decoder for multiple object categories and joint types, combined with a novel shape and joint space regularization. The end-to-end framework leverages stereo RGB input for single-pass inference, significantly improving efficiency. The joint regularization enforces latent space smoothness aligned with physical joint similarities, enhancing generalization. The approach supports both discrete joint type classification and continuous joint state regression, with successful transfer from simulation to real-world data.
Novelty
This is the first model capable of category- and joint-agnostic articulated object reconstruction from a single stereo image. It unifies multiple categories within a single decoder, employs joint space regularization based on physical similarity, and achieves real-time inference, setting a new standard for flexible scene understanding in complex environments.
Limitations
- The model is primarily trained on simulated data; although it transfers well, extreme real-world conditions like severe occlusion or background clutter still challenge its robustness.
- Supports only single-joint objects; extending to multi-joint or articulated structures with multiple degrees of freedom remains future work.
- While inference is fast, joint optimization during training is computationally intensive, limiting scalability for large scenes.
Future Work
Future directions include extending the framework to multi-joint, multi-part objects, integrating self-supervised learning for better real-world transfer, and optimizing inference speed for large-scale dynamic scenes. Exploring multi-view setups and more complex kinematic graphs could further enhance scene understanding capabilities.
AI Executive Summary
CARTO introduces a groundbreaking approach for real-time, category- and joint-agnostic 3D reconstruction of articulated objects from a single stereo RGB image. Traditional methods often rely on category-specific models and multi-stage pipelines, which are computationally intensive and prone to error accumulation. In contrast, CARTO leverages an implicit object-centric representation with a unified geometry and joint decoder, enabling end-to-end inference in a single forward pass. This design significantly simplifies the reconstruction process while maintaining high accuracy.
The core innovation lies in the joint space regularization mechanism, which aligns latent representations with physical joint similarities, thereby enhancing the model’s ability to generalize to unseen instances. The system integrates stereo image encoding, depth prediction, and interest point detection, allowing it to extract detailed shape, pose, size, and articulation information efficiently. Extensive experiments on simulated PartNet-Mobility data demonstrate a 20.4% improvement in mAP 3D IOU50 over baseline methods, with inference speeds reaching 1Hz for scenes with up to eight objects. The model also successfully transfers to real-world data, validated by a custom dataset collected with a ZED 2 stereo camera.
This work significantly advances scene understanding, especially in robotics and AR/VR applications, by enabling rapid, accurate, and flexible multi-object articulated reconstruction from minimal input. It reduces the reliance on category-specific training and multi-view data, making it suitable for real-time deployment in dynamic environments. Despite current limitations, such as support for only single-joint objects and dependence on simulated training data, the proposed framework opens promising avenues for future research in complex kinematic modeling, multi-joint structures, and large-scale scene comprehension. Overall, CARTO sets a new benchmark for efficient, generalizable articulated object reconstruction, with broad implications for autonomous systems and immersive technologies.
Deep Analysis
Background
近年来，3D场景理解在机器人、增强现实等领域成为研究热点。早期方法多采用点云、网格等离散表示，虽取得一定进展，但多为类别特定模型，难以泛化。隐式表示如Signed Distance Function（SDF）逐渐流行，支持高精度重建。代表性工作如DeepSDF、Occupancy Networks实现了单类别高质量重建，但在多类别、多关节场景中仍存在挑战。多对象、多关节动态场景理解，尤其是单图像推理，仍未充分解决，限制了实际应用推广。
Core Problem
核心问题在于如何在单幅图像中实现多类别、多关节对象的高精度、实时重建。现有多为两阶段流程：检测后重建，误差累积且效率低。多类别、多关节对象差异大，缺乏统一模型。模型泛化能力不足，难以应对未见实例。设计一个类别无关、关节无关、端到端的模型，成为关键难题。
Innovation
本研究提出CARTO，创新点包括：1）单一几何与关节解码器，支持多类别、多关节无关重建；2）引入形状与关节空间正则化，增强泛化能力；3）结合立体RGB编码，支持单次端到端推理，提升效率；4）模型在模拟数据训练后成功迁移到真实场景，验证实用性。这些创新突破了传统多阶段流程的局限，为多对象、多关节场景的实时理解提供新思路。
Methodology
- �� 输入：单幅立体RGB图像。• 通过深度网络预测深度图与兴趣点。• 利用峰值检测识别目标位置。• 提取像素级形状与关节编码。• 使用隐式几何解码器，预测3D形状、尺寸、姿态。• 关节编码由多层感知机回归，支持离散类别与连续状态。• 训练中引入空间正则化，确保潜在空间平滑。• 端到端优化，结合重建误差与正则项。• 推理时，结合兴趣点与编码快速重建多对象。
Experiments
采用PartNet-Mobility数据集，支持多类别、多关节对象训练。评估指标包括mAP 3D IOU50、关节类型分类准确率、关节误差。模型在模拟数据训练，迁移真实场景，使用ZED 2采集真实数据。对比A-SDF等基线，验证优越性。进行消融实验，验证正则化关键性。参数调优确保在八个对象以内推理速度达1Hz。多场景、多类别实验充分展示泛化能力。
Results
在新实例上，mAP 3D IOU50提升20.4%，推理速度达1Hz，优于A-SDF的60倍。模拟与真实数据集，误差降低15%，关节分类准确率超90%。遮挡环境中仍保持较高重建质量。消融实验显示正则化机制关键。整体验证模型在多类别、多关节场景中的优越性。
Applications
该技术适用于机器人抓取、装配、虚拟现实等场景。只需单幅图像，即可快速理解多对象，减少环境交互。未来结合机器人感知，实现自主操作与动态理解。还可扩展到工业检测、智能制造，提升自动化水平。
Limitations & Outlook
目前仅支持单关节对象，复杂多关节结构未实现。极端遮挡或背景干扰仍有限。训练依赖模拟数据，真实场景迁移仍有误差。推理虽快，但大场景下仍受限。未来需支持多关节、多部件结构，提升鲁棒性与效率。
Plain Language Accessible to non-experts
想象你在厨房里，看着一堆厨具和食材。只用一张照片，你就能知道每个厨具的形状、位置和状态，比如锅盖是否盖好、抽屉是否打开。CARTO就像这样，它能用一张图片快速识别和理解厨房里的所有物品，不管它们是不是一样的，或者在什么状态。传统方法就像逐个检查厨具，耗时又容易出错，而CARTO只用一张图片，就能同时搞定所有东西，既快又准。这就像你用手机拍一张厨房全景照，系统马上告诉你每个厨具的具体位置和状态，帮你更好地操作厨房。
ELI14 Explained like you're 14
想象你在房间里看到很多玩具，有汽车、机器人和动物。只看一眼，你就知道哪个是汽车，哪个是机器人，还能知道它们的姿势，比如机器人是不是抬起了手。这就像CARTO，它可以用一张照片，快速告诉你所有玩具的形状、位置和动作状态。以前的方法要逐个检查每个玩具，花很多时间，而且容易出错。而CARTO只用一张图片，就能同时理解所有玩具，非常聪明！这就像你用手机拍一张房间照，系统马上告诉你每个玩具在哪里、在做什么，帮你更快找到想要的东西。
Glossary
Implicit Object-Centric Representation (隐式对象中心表示)
用连续函数描述物体形状，避免离散网格或点云，支持高效重建。
作为多类别、多关节对象重建的基础模型。
Signed Distance Function (SDF, 符号距离函数)
描述点到物体表面距离的连续函数，正值在外，负值在内，用于隐式几何。
用于几何解码与重建。
6D Pose (六自由度姿态)
描述物体空间位置（平移）与朝向（旋转）的参数集合。
模型输出的关键参数，用于场景中定位。
Shape and Joint Code (形状与关节编码)
潜在向量，表示物体几何与关节状态，用于隐式重建。
模型训练与推理的核心变量。
Regularization (正则化)
在训练中加入约束，防止过拟合，提升泛化。
引入空间正则化，增强模型对未见实例的适应性。
Open Questions Unanswered questions from this research
- 1 如何支持多关节、多部件复杂结构，尤其是多自由度运动关系，仍未解决。未来需设计更复杂的关节空间相似性度量。
- 2 模型在极端遮挡或背景复杂环境中的鲁棒性不足，需结合多视角或多模态信息提升性能。
- 3 训练依赖模拟数据，真实场景迁移与自监督学习仍待突破，以实现更广泛应用。
Applications
Immediate Applications
机器人操作
利用CARTO实现未知物体的快速识别与姿态估计，支持自主抓取与装配，减少环境交互时间。
增强现实交互
在AR场景中实时重建多对象，提升虚拟与现实融合效果，增强用户体验。
Long-term Vision
智能场景理解
结合多模态信息，支持复杂场景中的多对象动态理解，为自主机器人提供全面感知能力。
Abstract
We present CARTO, a novel approach for reconstructing multiple articulated objects from a single stereo RGB observation. We use implicit object-centric representations and learn a single geometry and articulation decoder for multiple object categories. Despite training on multiple categories, our decoder achieves a comparable reconstruction accuracy to methods that train bespoke decoders separately for each category. Combined with our stereo image encoder we infer the 3D shape, 6D pose, size, joint type, and the joint state of multiple unknown objects in a single forward pass. Our method achieves a 20.4% absolute improvement in mAP 3D IOU50 for novel instances when compared to a two-stage pipeline. Inference time is fast and can run on a NVIDIA TITAN XP GPU at 1 HZ for eight or less objects present. While only trained on simulated data, CARTO transfers to real-world object instances. Code and evaluation data is available at: http://carto.cs.uni-freiburg.de