HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
HumanEgo uses entity-level hand-object interaction representation and flow matching to enable zero-shot robot learning from minutes of human egocentric videos, achieving 92.5% success.
Key Findings
Methodology
HumanEgo employs an interaction-centric representation (ICT) encoding relative poses of hands and objects, achieved via arm inpainting and 3D pose estimation. It leverages flow matching with dense auxiliary tasks—object motion, 2D trajectory, latent consistency—to enhance learning efficiency from limited data. The approach is hardware-agnostic, requiring no robot data, and enables cross-embodiment zero-shot transfer. The pipeline involves collecting egocentric videos with Aria glasses, preprocessing visual inputs to remove body bias, encoding spatial relationships into compact tokens, and training a flow-based policy with multi-task supervision.
Key Results
- On four real-world tasks, HumanEgo trained with only 30 minutes of human videos achieved an average success rate of 92.5%, with 75% success at just 15 minutes. It outperformed robot teleoperation (success rate 51.2%) by 41%, demonstrating high data efficiency. The policy generalized robustly across different robots (Trossen, Franka, UR10), environments, and camera setups, with success rates above 85% in out-of-distribution conditions.
- Comparison of data collection methods revealed human videos contain higher spatial density and trajectory diversity, enabling more effective learning. 8 minutes of human data surpassed 30 minutes of robot data in performance, highlighting the superior efficiency of human demonstrations.
- Cross-condition tests showed the policy maintained high success rates (>85%) despite variations in background, lighting, viewpoint, and object instances, confirming the strong generalization enabled by interaction-centric encoding.
Significance
This work advances robot learning by enabling rapid, data-efficient, and hardware-agnostic policy acquisition from minimal human demonstrations. It addresses core challenges of embodiment gap and data sparsity, providing a scalable solution for deploying robots in diverse real-world scenarios. The zero-shot transfer capability across different robots and environments paves the way for more adaptable autonomous systems, reducing reliance on extensive robot-specific data collection and fine-tuning. Such progress could revolutionize industrial automation, service robotics, and personal assistants, making robot deployment faster, cheaper, and more flexible.
Technical Contribution
The key innovation is the interaction-centric representation (ICT), which encodes hand-object spatial relationships invariant to viewpoint and embodiment, enabling effective transfer. The flow matching policy, combined with multi-density auxiliary supervision, allows fast, expressive action generation from limited data. The arm inpainting technique removes visual embodiment bias, and the multi-task training enhances scene understanding. This integrated approach surpasses prior methods relying on point clouds, goal-conditioned models, or diffusion-based policies, establishing a new paradigm for data-efficient robot learning.
Novelty
This is the first work to encode hand-object interactions explicitly at an entity level for zero-shot transfer from egocentric videos, bridging the embodiment gap without robot data. It combines flow-based action generation with dense supervision, achieving high performance with minimal demonstrations. Unlike prior methods focusing on point clouds or goal-conditioned trajectories, HumanEgo emphasizes the interaction geometry, enabling robust cross-embodiment generalization, marking a significant leap in robot imitation learning.
Limitations
- The approach depends on accurate 3D pose estimation; errors in perception can degrade performance, especially in cluttered or occluded scenes.
- Handling highly dynamic or occluded environments remains challenging, requiring further robustness improvements.
- Current reliance on high-quality perception modules and visual preprocessing limits scalability in extremely complex or real-time scenarios. Future work should focus on integrating more robust perception and adaptive learning mechanisms.
Future Work
Future directions include integrating reinforcement learning for fine-tuning policies in dynamic environments, expanding multi-robot collaboration capabilities, and enhancing perception robustness through multi-modal sensing. Exploring online adaptation and continual learning could further improve performance in long-term deployments, making robots more autonomous and versatile.
AI Executive Summary
HumanEgo introduces a novel framework for robot manipulation learning from minimal human egocentric videos, addressing key challenges of embodiment gap and data efficiency. Traditional robot learning methods rely heavily on task-specific demonstrations, which are costly and time-consuming. In contrast, HumanEgo leverages the rich manipulation demonstrations captured from human first-person videos, transforming them into a transferable, embodiment-invariant representation.
At the core of this approach is the interaction-centric representation (ICT), which encodes the spatial relationships between hands and objects in a compact, view-invariant manner. This is achieved through arm inpainting techniques that remove visual body differences, and by tracking 3D poses of hands and objects to generate entity-level tokens. These tokens serve as the state input for a flow matching policy, which predicts multi-modal actions efficiently. To maximize supervision from limited data, the system incorporates dense auxiliary objectives—predicting object trajectories, 2D visual traces, and scene dynamics—further boosting learning performance.
Experimental results demonstrate that with only 30 minutes of human demonstration, HumanEgo achieves an average success rate of 92.5% across four complex tasks, outperforming existing methods by a significant margin. Remarkably, the policy generalizes robustly across different robots, environments, and camera setups without retraining, exemplifying its zero-shot transfer capability. Even with just 15 minutes of data, it surpasses robot teleoperation methods trained on much longer datasets.
This work signifies a major step toward scalable, data-efficient robot learning. By harnessing the natural richness of human demonstrations, it reduces the barrier to deploying autonomous robots in diverse real-world scenarios. The approach’s simplicity, efficiency, and robustness open new avenues for industrial automation, service robots, and beyond. Future research will focus on improving perception robustness, online adaptation, and multi-robot collaboration, further pushing the boundaries of autonomous manipulation.
Deep Analysis
Background
机器人任务学习一直是人工智能和机器人学的核心问题。传统方法如行为克隆（Behavior Cloning）和强化学习（Reinforcement Learning）虽然取得一定进展，但都依赖大量的标注和示范，成本高昂且难以快速适应新任务。近年来，模仿学习和迁移学习成为研究热点，代表性工作包括GAIL、DALL·E等，但大多依赖机器人示范数据或大规模预训练。人类视频，尤其是第一视角的操控视频，因其低成本和丰富信息，逐渐成为替代方案。特别是头戴相机捕获的操控示范，提供了丰富的手-物交互信息，为机器人自主学习提供了新的可能。
Core Problem
现有方法在跨身体和环境迁移方面表现有限，主要受视觉差异和运动差异的限制。人体与机器人在外观和运动学模型上存在巨大差异，导致直接模仿效果差。少样本学习面临多模态信号稀疏的问题，如何从有限的人类示范中提取丰富的操控信息成为难点。此外，缺乏一种统一的表示方法来桥接不同身体和环境的差异，限制了模型的泛化能力。
Innovation
HumanEgo的创新点包括：1）提出实体级手-物交互表示（ICT），实现视角和身体差异的完全不变；2）引入Arm inpainting技术，消除视觉上的人体差异；3）采用流匹配（Flow Matching）策略，快速生成多模态动作；4）设计多密度辅助目标（物体运动、2D轨迹、潜在一致性），增强少样本学习能力。这些创新共同实现了从少量人类视频到高性能机器人策略的跨域迁移，突破了传统的视觉和运动差异限制。
Methodology
- �� 数据采集：使用Aria眼镜采集任务演示，30Hz，少量示范。
- �� 视觉预处理：用SAM分割手部，用LaMa inpainting消除人体身体差异，渲染虚拟夹持器和关键点。
- �� 空间关系编码：追踪手和物体的6-DoF姿态，利用ICT编码相对关系，确保视角不变。
- �� 动作生成：采用流匹配（Flow Matching）策略，条件化速度场，结合高效ODE求解器快速生成动作。
- �� 辅助目标：预测物体轨迹（LOM）、2D投影（L2D）和潜在一致性（LLC），丰富监督信号。
- �� 训练：端到端优化，利用多任务损失增强少样本学习能力。
Experiments
在四个真实任务（面包递送、杯子堆叠、浇水、调节桌子）中评估，比较多种零样本方法和机器人遥操作。使用成功率作为指标，验证模型在不同条件下的泛化能力。通过不同数据量（15-30分钟）分析学习效率，进行消融实验确认空间关系表示和辅助目标的重要性。模型在不同机器人和环境中实现无调优迁移，表现优异。
Results
HumanEgo在四个任务中平均成功率达92.5%，15分钟示范即可达到75%，优于机器人遥操作51.2%。在复杂任务如杯子堆叠和浇水中表现尤为突出，空间关系编码和多密度辅助目标显著提升性能。模型对不同机器人和环境条件具有强鲁棒性，成功率均在85%以上，验证了其泛化能力和实际应用潜力。
Applications
可广泛应用于工业自动化、家庭服务、医疗辅助等场景，特别适合缺乏大量机器人示范的任务。只需少量人类视频即可快速部署，降低成本，提高效率。未来结合自主强化学习，将实现更复杂环境下的自主操作。
Limitations & Outlook
目前依赖高质量的空间感知和姿态估计，感知误差会影响性能。复杂环境中的遮挡和动态变化仍是挑战，模型对感知系统的依赖较强。未来需提升感知鲁棒性和多模态融合能力，以应对更复杂的实际场景。
Plain Language Accessible to non-experts
想象一个厨房，厨师用手拿着各种工具做菜。传统机器人就像是个笨手笨脚的助手，只能模仿厨师的动作，但不能理解为什么要这样做。HumanEgo就像是一个聪明的助手，它不用看厨师的身体，只关注厨师手和工具的关系，知道什么时候需要用力、什么时候放松。它通过观察厨师用手操作，学会了做菜的关键步骤，然后可以自己去完成任务。这个方法就像是让机器人看懂了厨师的手势和工具的关系，而不是简单模仿动作，从而能在不同厨房、不同厨师之间自由工作。这就像是学会了做菜的“秘密配方”，不用每次都教它具体步骤，只要看几次就能自己做得很好。
ELI14 Explained like you're 14
想象你在厨房帮妈妈做饭。妈妈用手拿着锅铲炒菜，你看得很清楚，但你不会模仿她的每一个动作，因为每个人的手都不一样。HumanEgo就像是一个聪明的机器人，它不用模仿妈妈的手，而是学会了妈妈用手和锅、菜的关系。它观察到：当妈妈用力推锅时，锅会动；当她拿起菜时，菜的状态会变化。只要学会了这些关系，机器人就可以自己去炒菜，不用每次都教它具体怎么做。它用一种特别的方法，把手和菜、锅的关系变成一种“秘密语言”，这样无论在哪个厨房、用哪个锅，它都能做得很好。这就像是学会了厨房里的“潜规则”，不用看具体动作，只看关系，就能自己做菜了。
Glossary
Interaction-Centric Tokens (ICT) (交互中心标记)
一种用来表示手和物体空间关系的紧凑编码，确保视角不变，便于跨身体和环境迁移。它将空间关系转化为数值向量，捕捉操作状态。
在论文中，ICT用以描述手-物体的相对空间关系，是实现零样本迁移的核心表示。
Flow Matching (流匹配)
一种基于微分方程的动作生成策略，通过学习速度场，将初始随机噪声转化为目标动作轨迹，快速生成多模态动作。
论文中采用流匹配作为动作生成器，结合辅助目标实现高效少样本学习。
Arm Inpainting (手臂修复)
利用图像修复技术消除人体手臂在视觉上的差异，生成无人体的虚拟场景，减少视觉差异带来的迁移难题。
在视觉预处理阶段，确保机器人视觉输入的无身体差异。
Dense Auxiliary Objectives (密集辅助目标)
在训练中引入多种预测任务（如物体轨迹、2D投影、潜在状态），丰富监督信号，提升少样本学习效果。
增强模型对场景动态的理解，改善操控策略的泛化能力。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂环境中保持感知和动作的鲁棒性，仍是未来研究的重点。现有方法对遮挡和动态变化敏感，需要结合多模态感知和自适应机制加强系统的适应能力。
- 2 模型在多机器人协作和长时任务中的表现尚未充分验证，未来需扩展多智能体环境中的学习和迁移能力。
- 3 感知误差对空间关系编码的影响仍未完全解决，提升空间感知的精度和鲁棒性是关键。
Applications
Immediate Applications
工业自动化
利用HumanEgo快速从少量人类演示中学习装配、搬运等任务，降低机器人部署成本，提升生产效率。
家庭服务机器人
通过简单示范，让家庭机器人学会日常任务，如倒水、整理物品，无需大量机器人数据。
Long-term Vision
自主学习平台
构建通用机器人自主学习系统，利用人类视频不断扩展技能库，实现多任务、多环境自主操作。
Abstract
Human egocentric video captures rich manipulation demonstrations without any robot hardware, yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics. We present HumanEgo, a framework that bridges the embodiment gap by lifting each human demonstration to an entity-level representation of hand-object interaction, and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory. HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable. With only 30 minutes of human videos per task, HumanEgo achieves 92.5% average success across four real-world tasks (75% with just 15 minutes), outperforms matched-time robot teleoperation by 41%, and robustly transfers zero-shot across novel robots, cameras, and environments. We release HumanEgo as an easy-to-use, open-source framework for learning robot policies directly from human data: https://github.com/TX-Leo/HumanEgo