Learning Versatile Humanoid Manipulation with Touch Dreaming
Introduces Touch Dreaming-enhanced HTD model, achieving 90.9% success in humanoid contact-rich manipulation via multimodal Transformer.
Key Findings
Methodology
This work proposes the Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer integrating touch, vision, and proprioception. It employs a single-stage training with behavioral cloning and auxiliary touch-dream heads predicting future hand forces and tactile latents supervised by an EMA target encoder. The system combines a RL-based lower-body controller, VR data collection platform, and tactile sensing, enabling complex contact-rich tasks. The model predicts structured actions and auxiliary future tactile signals, regularizing the shared Transformer trunk to learn contact-aware dynamics. Experiments across five real-world tasks show a 90.9% relative success rate improvement over baselines, with latent-space tactile prediction outperforming raw tactile prediction by 30%.
Key Results
- HTD achieves an average success rate of 90.9% across five tasks, outperforming the baseline by approximately 90%. The latent tactile prediction head yields a 30% success rate increase over raw tactile prediction, demonstrating the effectiveness of touch dreaming. The system excels in precise insertion, deformable object handling, low-profile tool use, and bimanual manipulation, showing strong generalization.
Significance
This research addresses the longstanding challenge of stable, contact-aware humanoid manipulation in complex environments. By integrating multimodal perception and future tactile prediction, it significantly enhances the robot’s dexterity and robustness, paving the way for autonomous systems in industrial, service, and medical applications. The approach offers a new paradigm for end-to-end learning of contact-rich behaviors, reducing reliance on handcrafted rules and multi-stage training, thus advancing the field toward more intelligent and adaptable robots.
Technical Contribution
Key innovations include the introduction of touch dreaming as an auxiliary task within a single-stage Transformer framework, leveraging EMA-supervised latent tactile prediction to avoid complex pretraining. The system fuses vision, tactile, and proprioceptive data into a unified latent space, enabling contact-aware dynamics modeling. This approach surpasses prior methods by integrating predictive touch into end-to-end policies, facilitating versatile, high-precision humanoid manipulation without multi-stage pipelines.
Novelty
This is the first work to embed future tactile prediction directly into a unified Transformer-based humanoid manipulation policy, combining latent space modeling with multimodal fusion. Unlike previous approaches relying solely on vision or raw tactile signals, this method leverages predictive touch as a regularizer, significantly improving contact-awareness and generalization in contact-rich tasks, representing a novel integration of touch dreaming into end-to-end robot learning.
Limitations
- The model’s robustness in highly dynamic or unpredictable environments remains limited, especially under extreme contact variations or high-speed interactions.
- Training requires extensive simulation and real-world data, which incurs high costs and limits scalability.
- Inference speed is constrained by Transformer complexity, impacting real-time deployment in fast-paced scenarios.
Future Work
Future efforts will focus on enhancing robustness to extreme contact conditions, integrating online adaptation and self-supervised learning to improve generalization. Additionally, optimizing Transformer architectures for faster inference and reducing data requirements will be key to deploying this system in industrial and service robots. Exploring autonomous exploration and active learning strategies could further improve contact modeling and task versatility.
AI Executive Summary
Humanoid robots are increasingly expected to perform complex, contact-rich manipulation tasks in real-world environments. However, existing systems often struggle with stability, dexterity, and contact awareness, especially under frequent contact changes. Traditional approaches rely heavily on vision and predefined rules, which limit adaptability and robustness. To address these challenges, this work introduces the Humanoid Transformer with Touch Dreaming (HTD), a novel multimodal deep learning framework that integrates tactile sensing, vision, and proprioception within a single Transformer architecture.
The core innovation lies in the touch dreaming mechanism, where the model predicts future hand forces and tactile latents, supervised by a stable EMA target encoder. This auxiliary task regularizes the shared latent space, enabling the robot to develop contact-aware representations crucial for dexterous manipulation. The system combines a reinforcement learning-based lower-body controller, VR-based data collection, and a unified perception-action pipeline, allowing for efficient demonstration collection and policy training.
Experimental results across five diverse real-world tasks—ranging from precise insertions to tool-mediated contact—demonstrate a remarkable success rate of 90.9%, far surpassing previous baselines. Notably, the latent-space tactile prediction head outperforms raw tactile prediction by 30%, validating the effectiveness of the touch dreaming approach. These findings highlight the potential of predictive multimodal learning to revolutionize humanoid manipulation, making robots more versatile, stable, and contact-aware.
While promising, the approach faces limitations such as high training costs, inference speed constraints, and challenges in extreme contact scenarios. Future research will aim to improve robustness, efficiency, and adaptability, paving the way for autonomous robots capable of complex, contact-rich tasks in industrial, medical, and service domains. Overall, this work marks a significant step toward intelligent, dexterous humanoid robots capable of operating seamlessly in unstructured environments.
Deep Dive
Glossary
Transformer (变换器)
一种深度学习模型架构，擅长处理序列数据，利用自注意力机制捕获长距离依赖。
用于融合多模态信息并建模动态接触关系。
触觉梦境 (Touch Dreaming)
一种预测未来触觉信号的机制，通过潜在空间表示增强接触感知。
模型在训练中预测未来手关节力和触觉潜在表示。
EMA目标编码器 (EMA Target Encoder)
利用指数移动平均维护的稳定触觉潜在目标，用于监督模型学习。
提供稳定的触觉目标，避免多阶段预训练。
行为克隆 (Behavioral Cloning)
模仿示范行为的监督学习方法，将专家示范映射到策略。
用于训练HTD模型的策略。
多模态融合 (Multimodal Fusion)
结合视觉、触觉和本体感知信息，形成统一表征的技术。
模型融合多源信息以增强感知能力。
Open Questions Unanswered questions from this research
- 1 当前模型在极端动态环境下的鲁棒性不足，未来需结合自主探索和在线学习提升适应性。
- 2 训练成本高，数据采集与标注依赖大量模拟与真实数据，限制推广。
- 3 推理速度受Transformer架构限制，实时应用仍需优化。
Applications
Immediate Applications
工业装配自动化
机器人可在复杂装配场景中实现高精度插入和微调，减少人工干预，提升生产效率。
医疗辅助机器人
用于精密手术或康复训练，机器人能感知细微接触变化，提供安全可靠的操作。
Long-term Vision
自主服务机器人
未来机器人能自主完成家务、护理等多样任务，具备高度接触感知与适应能力。
Abstract
Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, end-effector dexterity, and contact-aware interaction under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based lower-body controller that serves as the stability backbone for whole-body execution during complex manipulation. Building on this controller, we develop a VR-based whole-body humanoid data collection system that integrates dexterous hands and tactile sensing for contact-rich manipulation. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder-decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, with tactile-latent targets provided by an exponential moving average target encoder without requiring a separate tactile pretraining stage. This encourages the policy to learn contact-aware representations for dexterous manipulation. Across five real-world contact-rich tasks, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline for each task. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that our touch-dreaming-enhanced learning system enables versatile, high-dexterity humanoid manipulation in the real world. More information and open-source materials are available at humanoid-touch-dream.github.io.