TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding
TACO constructs a large-scale bimanual tool-action-object dataset supporting generalization tasks with 2.5K sequences and 5.2M frames.
Key Findings
Methodology
TACO employs a multi-view synchronized data collection system combining optical motion capture and markerless vision, automatically annotating hand and object meshes, segmentation, and action labels. The dataset includes 131 tool types, 20 object categories, and 2.5K sequences with 5.2 million frames, enabling cross-category generalization. The authors benchmark three tasks: compositional action recognition, motion forecasting, and cooperative grasp synthesis, using models like AIM and CACNF for evaluation.
Key Results
- In compositional action recognition, CACNF achieved 39.33% Top-1 accuracy on the S4 test set (unseen tool categories and interactions), significantly outperforming baseline models, demonstrating strong generalization to novel interactions.
- For motion forecasting, models trained on TACO data reduced hand position errors (MPJPE) by over 20% compared to baselines, showing high-fidelity dynamic capture and prediction capabilities.
- In grasp synthesis, models leveraging hand-object meshes successfully generated natural grasping motions on unseen object categories, illustrating the dataset's effectiveness for complex interaction modeling.
Significance
This dataset advances the field by enabling research on multi-hand, multi-object interactions in realistic scenarios, addressing the limitations of prior datasets focused on single-hand or single-object tasks. It fosters development of models capable of generalizing to unseen objects and interactions, crucial for robotics, AR/VR, and human-robot collaboration, ultimately pushing towards more intelligent, adaptable systems.
Technical Contribution
The paper introduces an automated multi-view data acquisition pipeline integrating optical motion capture and vision-based annotation, ensuring high-precision 3D meshes and labels. The dataset covers diverse tools and objects, supporting multi-task benchmarks. The authors also propose comprehensive evaluation protocols for generalization, providing insights into current model limitations and future directions.
Novelty
TACO is the first large-scale, real-world, multi-object, bimanual interaction dataset with automatic annotation, spanning diverse tools and objects, supporting multi-task learning and cross-category generalization. Its automated pipeline significantly improves data collection efficiency over prior manual methods, setting a new standard for complex interaction datasets.
Limitations
- The dataset is primarily collected in controlled environments, limiting variability in real-world backgrounds and lighting conditions. This may affect model robustness in unconstrained settings.
- Handling highly complex interactions, such as multi-hand coordination or deformable objects, remains challenging and underrepresented.
- High-cost equipment and multi-view setups restrict large-scale deployment, necessitating future efforts to reduce hardware dependency.
Future Work
Future research will expand data diversity to include more complex, real-world scenarios, integrating soft-body interactions and multi-user cooperation. Developing self-supervised learning approaches to reduce annotation costs and improve model robustness in varied environments is also a priority. Additionally, real-time interaction understanding and generation will be explored to enable more natural human-robot collaboration.
AI Executive Summary
TACO represents a groundbreaking step in understanding complex bimanual tool-object interactions. By combining multi-view synchronized data collection with an automated annotation pipeline, it delivers a comprehensive dataset of 2.5K sequences and over 5 million frames, capturing diverse tool and object categories in realistic scenarios.
This dataset addresses a critical gap in the field, where prior efforts focused mainly on single-hand or single-object tasks, limiting models’ ability to generalize. TACO’s rich annotations—including precise 3D hand and object meshes, segmentation masks, and action labels—enable detailed analysis and benchmarking across multiple tasks.
The authors evaluate three core tasks: compositional action recognition, motion forecasting, and cooperative grasp synthesis. Results demonstrate that models trained on TACO significantly outperform baselines in unseen tool categories and interaction triplets, with the best models achieving 39.33% Top-1 accuracy in the most challenging scenarios. These findings highlight the dataset’s potential to foster robust, generalizable models for real-world applications.
Beyond technical achievements, TACO paves the way for advancements in robotics, AR/VR, and human-computer interaction by providing a high-quality, scalable resource for complex interaction understanding. Its automated data collection process reduces manual effort, enabling future expansion and more diverse scenarios. While current limitations include environmental variability and hardware costs, ongoing efforts aim to address these challenges, broadening the impact of this work in real-world settings.
Deep Analysis
Background
手部交互研究从二维动作识别逐步发展到三维网格重建，代表性数据集如HO3D、DexYCB等，主要关注单手操作。近年来，多手、多物体复杂交互成为研究热点，但缺乏大规模、真实场景数据支持，限制模型泛化能力。现有数据多偏向静态或单任务，难以满足多场景、多任务需求。TACO应运而生，旨在弥补这些不足，推动多手多物体交互理解的研究。
Core Problem
当前数据集在多手、多物体交互方面规模不足，难以支持泛化研究。多手操作的动态复杂性、工具多样性和交互多样性带来采集和标注难题。缺少高质量、多视角、自动化标注流程，限制模型在真实复杂场景中的应用。解决这些瓶颈，提升模型泛化能力，成为研究重点。
Innovation
提出全自动多视角同步采集系统，结合光学捕捉与无标记视觉，确保高精度手部与物体的3D重建。构建涵盖多类别、多场景、多任务的数据集，支持跨类别泛化。设计多任务基准，涵盖动作识别、运动预测与合成，推动多手多物体交互理解技术发展。引入行为三元组标注，丰富交互语义信息，提升模型表达能力。
Methodology
- �� 多视角同步采集：利用12个工业相机与光学捕捉设备，采集手部与物体运动。
- �� 自动标注流程：结合标记点优化、手部关键点检测、手-物体分割与标记去除，确保高质量数据。
- �� 3D网格重建：利用工业扫描仪获取目标物体高细节模型。
- �� 多任务学习：在识别、预测与合成任务中，利用深度神经网络（如AIM、CACNF）进行训练。
- �� 数据增强：通过多视角、多场景变化，提升模型泛化能力。
Experiments
采用20类目标、131个工具类别，进行多任务性能评估。设置不同泛化场景（如新工具类别、未见交互组合），用Top-1、Top-5准确率、MPJPE等指标衡量模型表现。对比多种模型（如AIM、CACNF），进行消融分析，验证数据集的有效性与模型的泛化能力。
Results
在组合动作识别任务中，CACNF在最具挑战的S4集达到Top-1准确率39.33%，优于传统模型。运动预测中，模型在手部位置误差方面比基线提升20%以上。合作合成中，基于手-物体网格的生成模型在未见物体类别上实现自然交互，验证了数据的多样性与实用性。
Applications
可应用于机器人自主操作、虚拟现实交互、手势识别等场景。高质量的多手多物体数据支持训练更智能的交互模型，推动自动化与人机协作技术发展。
Limitations & Outlook
目前数据主要在受控环境采集，真实场景中的复杂背景、光照变化尚未充分覆盖。多手操作、软体变形等复杂交互场景支持有限。高成本设备和多视角设置限制大规模应用，未来需优化硬件依赖。
Plain Language Accessible to non-experts
想象你在厨房里做饭，双手配合得天衣无缝：一只手拿锅，另一只手用锅铲搅拌。这种动作很自然，但背后其实很复杂。机器人要学会这些动作，就像你学做菜一样，需要看很多视频，理解每个工具的用法。TACO就像是给机器人拍的厨房录像，里面有人用不同的工具做饭，动作清晰又丰富。通过学习这些录像，机器人将来可以自己用新工具做菜，甚至帮你洗碗、搬东西。它们变得越来越聪明，能帮你做各种家务事，就像你有个超级帮手一样。
ELI14 Explained like you're 14
想象你在学校的科学课上拼装模型，你用不同的零件拼出各种东西。有时候你用积木搭桥，有时候用拼图拼车。这些动作都需要你知道每个零件的作用和怎么拼。TACO就像是给机器人拍的超级详细的录像，里面的人用各种工具做不同的事情，比如用锤子钉钉子、用剪刀剪纸。机器人可以通过看这些录像学会用新工具做新事。它们学会了怎么用手、怎么抓东西，还能预测下一步会怎么动，甚至帮你设计出漂亮的模型。虽然还不能完全像人一样灵活，但这一步让机器人更聪明、更会帮忙啦！
Abstract
Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities. However, existing technical approaches for analyzing and synthesizing hand-object manipulation are mostly limited to handling a single hand and object due to the lack of data support. To address this, we construct TACO, an extensive bimanual hand-object-interaction dataset spanning a large variety of tool-action-object compositions for daily human activities. TACO contains 2.5K motion sequences paired with third-person and egocentric views, precise hand-object 3D meshes, and action labels. To rapidly expand the data scale, we present a fully automatic data acquisition pipeline combining multi-view sensing with an optical motion capture system. With the vast research fields provided by TACO, we benchmark three generalizable hand-object-interaction tasks: compositional action recognition, generalizable hand-object motion forecasting, and cooperative grasp synthesis. Extensive experiments reveal new insights, challenges, and opportunities for advancing the studies of generalizable hand-object motion analysis and synthesis. Our data and code are available at https://taco2024.github.io.