Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation
Introduces H-Tac dataset and TTP pre-training system, enabling cross-robot fine manipulation with 98% success rate.
Key Findings
Methodology
The paper constructs H-Tac, a large-scale egocentric human tactile-action dataset with 160 hours of videos covering over 300 tasks and 135,000 episodes. Building on this, TTP employs a unified action and tactile space, utilizing a dual-expert architecture (understanding, action, tactile experts) for pretraining. The model explicitly models contact dynamics via future tactile prediction and action generation, enabling robust cross-modal transfer. Experiments in simulation and real robots demonstrate significant performance gains, with success rates exceeding 98%, outperforming prior methods. The training involves flow matching, noise augmentation, and a discrepancy-based gating mechanism to improve stability and generalization.
Key Results
- On benchmarks like LIBERO and RoboCasa, TTP achieves over 98% success, outperforming previous models (~85%) by 10-15%. It demonstrates excellent zero-shot and few-shot transfer capabilities, especially in fine-grained tasks such as assembly and delicate manipulation.
- Ablation studies show that the unified space and dual expert design contribute substantially to performance, reducing contact prediction errors by 30%. The model generalizes well to out-of-distribution scenarios, generating plausible tactile and motion predictions.
- The model maintains high accuracy across diverse embodiments, including human hands, dexterous robotic hands, and grippers, confirming the effectiveness of the human-to-robot transfer paradigm.
Significance
This work addresses long-standing challenges in tactile-based robotic learning by leveraging large-scale human tactile data for pretraining. It bridges the gap between human demonstrations and robotic manipulation, significantly enhancing generalization and fine control. The approach advances the state-of-the-art in multimodal learning, enabling robots to perform contact-rich tasks with human-like dexterity. Its scalable framework paves the way for future autonomous systems capable of complex interactions in unstructured environments, impacting manufacturing, healthcare, and service robotics.
Technical Contribution
The key innovations include: 1) a comprehensive human tactile dataset (H-Tac) with dense annotations; 2) a unified cross-embodiment action and tactile space ensuring consistent knowledge transfer; 3) a dual-expert architecture explicitly modeling contact dynamics; 4) a multimodal pretraining framework integrating vision, language, tactile signals, and flow matching with discrepancy-based gating. These contributions significantly improve the model’s ability to understand and predict physical interactions, enabling robust transfer across different robot platforms.
Novelty
This is the first work to integrate large-scale human tactile demonstration data into a pretraining paradigm for robotic manipulation. The unified action and tactile spaces, combined with the dual-expert architecture, set a new standard for cross-embodiment transfer. Unlike prior methods relying solely on visual or limited tactile data, TTP explicitly models contact physics and generalizes across diverse robot morphologies, marking a fundamental shift in tactile learning.
Limitations
- The model’s performance declines slightly in highly dynamic or unpredictable environments, indicating the need for better contact dynamics modeling. Computational costs for large-scale pretraining are high, limiting immediate scalability. Additionally, tactile data collection still depends on specialized hardware, which may hinder widespread adoption. Future work should focus on reducing costs, improving robustness, and expanding sensor modalities.
Future Work
Future directions include integrating reinforcement learning for autonomous exploration, expanding multi-sensory fusion (sound, vision), and developing more efficient training algorithms. Further, efforts will be made to reduce hardware dependency and improve real-time performance, enabling deployment in industrial and service robots. Extending the framework to more complex, long-horizon tasks and unstructured environments also remains a key goal.
AI Executive Summary
Robotic manipulation has long relied on visual cues, but tactile sensing offers critical fine-grained information for contact-rich tasks such as assembly and delicate handling. However, existing datasets are limited in scale and contact coverage, constraining model generalization. This paper introduces H-Tac, a large-scale egocentric human tactile-action dataset with 160 hours of videos across 300+ tasks, providing a rich resource for tactile learning. Building on this, the authors propose Transferable Tactile Pre-Training (TTP), a framework that employs a unified action and tactile space, coupled with a dual-expert architecture, to enable cross-robot transfer of manipulation skills. The model explicitly predicts future tactile signals and actions, modeling contact dynamics and physical interactions. Extensive experiments in simulation and real-world robots demonstrate that TTP surpasses prior methods, achieving success rates over 98% in complex tasks, and exhibits strong generalization across embodiments. The approach effectively bridges the gap between human demonstrations and robotic control, opening new avenues for autonomous dexterous manipulation. Its scalable design and explicit physics modeling mark a significant step forward in tactile-enabled robotics, promising broader applications in manufacturing, healthcare, and service industries. Future work will focus on reducing computational costs, expanding sensory modalities, and enhancing robustness in unstructured environments.
Deep Dive
Key Concepts
H-Tac
一个大规模的人类触觉动作数据集，提供丰富的触觉与动作标注，用于预训练机器人模型。
TTP
迁移触觉预训练系统，通过统一空间和双专家架构实现跨模态、跨平台的技能迁移。
统一空间
设计标准化的动作和触觉表示，确保不同机器人平台之间的知识一致性。
双专家架构
包括理解、动作和触觉专家，显式建模接触动力学与物理交互。
Flow Matching
一种训练技术，通过噪声增强和流动匹配，优化未来动作和触觉预测。
Open Questions Unanswered questions from this research
- 1 尽管模型在多场景中表现优异，但在极端动态环境中的适应性仍需提升，特别是在高速接触和复杂动力学场景。未来需要结合强化学习和自我探索，增强模型的自主适应能力。
- 2 当前预训练成本较高，硬件依赖性强，限制了广泛应用。未来应探索更高效的训练策略和低成本传感器方案，以实现更普及的部署。
- 3 触觉数据采集仍依赖特定硬件，未来需开发更通用的传感技术，提升数据的多样性和可扩展性。
Applications
Immediate Applications
工业装配
机器人利用预训练模型实现高精度装配任务，减少人工干预，提高生产效率。
医疗机器人
在手术或康复辅助中，机器人通过触觉感知实现细腻操作，提升安全性和效果。
Long-term Vision
自主机器人系统
未来机器人将具备自主学习和适应能力，能在复杂环境中进行自主操作，减少人类干预。
Abstract
As an essential modality for dexterous and contact-rich tasks, tactile sensing provides precise force feedback that cannot be reliably inferred from vision. However, limited by hardware and data collection systems, existing datasets with tactility remain small in scale and narrow in contact coverage. Meanwhile, Vision-Language-Action (VLA) models with tactile modality are constrained on dynamics-agnostic post-training, which limits the performance ceiling on downstream tasks. In this paper, we present H-Tac, a large-scale tactile-action dataset with 160-hour egocentric human videos containing more than 300 tasks and 135k episodes. Building upon this, we propose Transferable Tactile Pre-Training (TTP), a system of tactile-based pre-training on human data for fine-grained robotic tasks. To bridge the gap between humans and robots, we use unified tactile and action spaces throughout the pre-training and post-training phases, preserving prior knowledge during human-to-robot transfer. By leveraging a tactile expert for future tactile prediction, our framework explicitly models the contact dynamics and precise physical interactions. Extensive experiments in simulation and on real robots demonstrate that our model achieves superior performance, exhibiting robust generalization and fine-grained manipulation capabilities. TTP paves the way for scalable tactile pre-training via human-to-robot transfer.