EgoZero: Robot Learning from Smart Glasses
EgoZero leverages egocentric human demonstrations captured via Project Aria glasses to learn robot manipulation policies with zero robot data, using point cloud-based state-action representations.
Key Findings
Methodology
EgoZero employs data collected from human demonstrations wearing Project Aria smart glasses, extracting 3D point cloud representations of objects and hands via triangulation and monocular tracking. It leverages models like Grounding DINO for object detection and DIFT for point tracking, integrating these into a unified shape-agnostic state space. The system trains a Transformer-based closed-loop policy through behavior cloning, minimizing the negative log-likelihood of predicted actions against demonstration data. Data augmentation techniques, including random rotations and translations, improve generalization. During inference, the robot uses an iPhone to capture the scene, reconstructs the environment's 3D points, and applies the learned policy to execute manipulation tasks, achieving high success rates without any robot-specific training data.
Key Results
- The system successfully transfers learned policies to a Franka Panda robot, achieving an average success rate of 70% across 7 manipulation tasks such as opening oven doors, placing bread, sweeping, erasing whiteboards, sorting fruits, folding towels, and inserting books. Each task required only 20 minutes of human demonstration data, collected in diverse environments with varying object poses and backgrounds. The policies demonstrated robust zero-shot generalization to unseen object configurations, camera viewpoints, and environmental conditions. Ablation studies confirmed the importance of point cloud triangulation, data augmentation, and target recognition models for performance. The results significantly outperform baseline vision prediction and affordance-based methods, highlighting the effectiveness of the shape-agnostic point cloud representation and closed-loop policy learning.
- The experimental results validate that high-fidelity 3D point cloud representations derived from egocentric visual data can enable effective zero-shot transfer, reducing dependence on robot-collected data and calibration. The approach demonstrates strong robustness against out-of-distribution scenarios, including different camera angles, object instances, and environmental backgrounds, emphasizing its potential for scalable real-world robot learning. The success across diverse tasks underscores the method's versatility, paving the way for autonomous robots capable of learning from natural human behaviors in unstructured environments.
Significance
This research marks a significant breakthrough in robot learning, enabling zero-shot manipulation skills transfer solely from in-the-wild human demonstrations without any robot data. It addresses longstanding bottlenecks related to data collection costs, environment-specific calibration, and limited generalization of existing methods. By harnessing natural human behaviors captured via simple wearable devices, EgoZero offers a scalable, cost-effective pathway toward building adaptable, intelligent robots capable of functioning in real-world settings like homes, factories, and disaster zones. The approach reduces reliance on labor-intensive data collection and calibration, democratizing robot learning and accelerating deployment in diverse applications. Its ability to generalize across objects, scenes, and viewpoints opens new horizons for autonomous systems that learn continuously from everyday human activities.
Technical Contribution
EgoZero introduces a novel shape-agnostic state-action representation based on 3D point clouds, circumventing the limitations of depth sensors and multi-camera calibration. It combines monocular visual tracking with triangulation over camera trajectories to localize objects without requiring explicit depth data. The system employs models like Grounding DINO for object detection and DIFT for point tracking, integrating these into a unified spatial framework. The core learning component is a Transformer-based closed-loop policy trained via behavior cloning, with extensive data augmentation to improve robustness. This architecture enables zero-shot transfer from natural human demonstrations to robotic execution, even in unseen environments. The approach also advances the field by demonstrating the effectiveness of minimal data collection (20 minutes per task) and the importance of shape-agnostic, geometry-based representations for generalization.
Novelty
This work is the first to achieve zero-shot transfer of manipulation skills from unstructured, in-the-wild human demonstrations without any robot data. Unlike prior approaches relying on multi-camera calibration, external depth sensors, or extensive robot-environment interaction, EgoZero leverages monocular visual cues and triangulation to reconstruct 3D point clouds. Its shape-agnostic state-action space allows for seamless generalization across object categories, poses, and environments. The integration of models like Grounding DINO and DIFT with a Transformer policy for closed-loop control represents a significant methodological innovation, enabling robust, data-efficient learning from natural human behaviors. This paradigm shift opens new avenues for scalable, naturalistic robot training.
Limitations
- The approach assumes static objects during demonstrations, limiting applicability in dynamic or cluttered scenes where objects or humans move. Handling such scenarios requires real-time tracking and adaptive modeling.
- Triangulation relies on sufficient camera movement and environment stability; in cases of limited motion or occlusion, spatial localization accuracy degrades, affecting policy performance.
- Hand keypoint detection models like HaMeR can struggle with occlusions or complex gestures, impacting the accuracy of action point extraction and subsequent manipulation success.
Future Work
未来的研究将集中于在动态环境中实现实时点云重建和目标追踪，结合深度传感器或立体视觉技术，提升空间定位的鲁棒性。同时，将探索多模态感知融合（如触觉、力觉）以增强操作的稳定性和精度。系统还将扩展到多手、多目标场景，支持多任务在线学习和自适应能力，推动机器人自主学习的普及。此外，研究者还计划优化点云三角测量算法，减少对环境静止的依赖，提升在复杂、动态场景中的应用能力。
AI Executive Summary
在机器人技术不断追求更高自主性和适应性的今天，如何让机器人在真实世界中自主学习复杂操控技能，成为科研界的核心难题。传统方法依赖大量机器人交互数据或在受控环境中进行训练，成本高昂且难以推广到多样化场景。本文提出的EgoZero系统，通过利用日常生活中人类佩戴的Project Aria智能眼镜采集的野外演示数据，开创了机器人零样本学习的新路径。
EgoZero的核心创新在于，将人类观察到的视觉信息转化为三维点云表示，利用三角测量和目标识别模型实现空间定位，避免了深度传感器的限制。系统通过Grounding DINO识别目标，DIFT追踪点云轨迹，结合Transformer模型进行闭环策略学习，训练出具有强泛化能力的操控策略。实验结果显示，该方法在7个不同任务中平均成功率达70%，每个任务仅需20分钟的人类演示，便能实现零样本迁移。
这些策略在未见过的环境、不同摄像头角度和物体配置中表现出优异的鲁棒性，验证了点云空间表示和数据增强的有效性。相较于传统视觉预测和轨迹模型，EgoZero在样本效率和泛化能力方面具有显著优势。这一突破不仅降低了机器人自主学习的门槛，也为未来在家庭、工业、救援等场景中的应用提供了坚实基础。
从技术角度看，EgoZero提出了基于点云的形态无关状态空间和闭环Transformer策略，创新性地解决了深度估计误差和迁移难题。其方法的普适性和高效性，为机器人自主学习提供了全新的框架，推动了从人类日常行为到机器人自主操控的转变。未来，系统将结合多模态感知、动态场景和多手操作，进一步提升适应性和实用性，开启机器人自主学习的新时代。
Deep Analysis
Background
机器人自主操控能力的提升一直是人工智能和机器人学的核心目标。早期方法主要依赖于精确的传感器和手工设计的操作模板，但在复杂、多变的真实环境中，这些方法面临诸多挑战。随着深度学习的发展，模仿学习和行为克隆成为研究热点，代表性工作如DAGGER、GAIL等，依赖大量机器人交互数据，训练高性能策略。然而，这些方法在实际应用中受到数据采集成本高、环境适应性差的限制。近年来，利用人类演示进行学习的研究逐渐兴起，采用虚拟现实、手势识别等技术实现人机交互，但仍需复杂硬件或多摄像头校准。点云和空间几何表示逐步成为解决方案的焦点，尤其是在单目视觉和有限硬件条件下实现空间重建方面取得了一定突破。本文所在的研究背景正是围绕如何从自然、未结构化的野外环境中提取稳定的空间表示，并实现跨形态、跨场景的迁移。
Core Problem
当前机器人学习在野外环境中的迁移能力不足，主要原因在于缺乏高效、鲁棒的空间表示和泛化策略。传统深度学习模型依赖于深度传感器或多视角校准，难以应对单目视觉深度估计的误差和环境变化。人类演示数据丰富，但如何将其转化为机器人可执行的动作，尤其是在没有机器人交互数据的情况下，成为一大难题。具体而言，如何从非结构化、动态、多视角的视觉数据中提取稳定的空间点云，成为技术瓶颈。解决这些问题，要求提出一种新颖的空间表示和学习框架，能够在有限演示数据下实现跨场景、跨形态的零样本迁移。
Innovation
EgoZero的创新主要体现在：首先，提出基于点云的形态无关状态空间，将二维视觉信息转化为三维点云，避免深度估计误差，增强空间表达的鲁棒性。其次，利用Grounding DINO和DIFT模型实现目标识别与点云追踪，结合三角测量技术，在无多摄像头校准条件下，准确定位目标空间位置。第三，采用Transformer模型作为闭环策略核心，结合行为模仿学习，实现从少量野外演示到复杂操控任务的强泛化。第四，系统引入多种数据增强技术，包括随机旋转、平移和点云扰动，提升模型在未见场景中的适应能力。这些创新共同构建了一个无需机器人数据、可在真实环境中快速迁移的自主学习框架，极大降低了机器人自主学习的门槛。
Methodology
- �� 数据采集：使用Project Aria智能眼镜采集野外人类演示，包括RGB图像、手部6DoF姿态和相机外参。• 空间表示：通过Grounding DINO识别目标，利用DIFT跟踪目标的UV坐标，结合相机外参进行三角测量，重建目标在空间中的点云。• 手部关键点提取：采用HaMeR模型获得手部21关键点，结合Aria手势信息，校正手部空间位置。• 点云融合：将目标点云和手部点云合成为统一的空间表示，作为状态输入。• 策略训练：利用Transformer模型，行为模仿学习最小化预测动作与演示动作的负对数似然，加入数据增强和噪声扰动，提升泛化能力。• 推理过程：在实际机器人上，利用iPhone摄像头捕获场景，进行点云重建和目标识别，利用预先校准的变换将策略输出映射到机器人空间，控制机器人执行任务。
Experiments
实验在Franka Panda机械臂上进行，采用七个不同操控任务，包括开门、放面包、扫地、擦白板、分水果、折毛巾和放书。每个任务采集100个演示，环境多样，物体位置变化显著。模型训练仅用演示数据，测试在未见过的环境和物体配置中，成功率平均达70%。对比基线模型（如视觉预测和开环轨迹模型），EgoZero在成功率和样本效率方面表现优越。通过消融实验验证点云三角测量、数据增强和目标识别模型的重要性。结果显示，该方法在不同摄像头视角、物体姿态和背景下均保持稳定性能，验证了其强大的泛化能力。
Results
EgoZero在七个任务中平均成功率达70%，显著优于传统视觉预测和基于轨迹的模型，且仅用20分钟演示数据。模型在不同摄像头角度、物体位置和背景条件下表现出优异的鲁棒性，验证了点云空间表示的有效性。消融实验显示，去除三角测量或数据增强会显著降低成功率，证明其关键作用。模型还能在超出训练数据分布的场景中保持较高成功率，展示了极强的泛化能力。这些结果表明，从野外人类演示中学习机器人操控策略成为可能，为机器人自主学习开辟了新路径。
Applications
该方法适用于家庭、工业、服务机器人等多场景，尤其在缺乏机器人交互数据的情况下，利用自然采集的人类演示实现自主操控。只需佩戴智能眼镜，无需复杂校准或多摄像头设置，即可在野外环境中采集数据并训练策略。未来，结合多模态感知和在线学习，将使机器人在动态环境中自主适应和优化操作策略，推动机器人在实际应用中的普及。
Limitations & Outlook
该方法假设环境静止且物体不移动，难以应对动态场景。三角测量对摄像头运动和遮挡敏感，可能导致点云定位误差。手部关键点检测模型在遮挡或复杂手势下表现不佳，影响策略精度。未来需引入多视角或深度传感器，提升空间定位鲁棒性，并扩展到动态、多手、多目标场景。
Abstract
Despite recent progress in general purpose robotics, robot policies still lag far behind basic human capabilities in the real world. Humans interact constantly with the physical world, yet this rich data resource remains largely untapped in robot learning. We propose EgoZero, a minimal system that learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses, $\textbf{and zero robot data}$. EgoZero enables: (1) extraction of complete, robot-executable actions from in-the-wild, egocentric, human demonstrations, (2) compression of human visual observations into morphology-agnostic state representations, and (3) closed-loop policy learning that generalizes morphologically, spatially, and semantically. We deploy EgoZero policies on a gripper Franka Panda robot and demonstrate zero-shot transfer with 70% success rate over 7 manipulation tasks and only 20 minutes of data collection per task. Our results suggest that in-the-wild human data can serve as a scalable foundation for real-world robot learning - paving the way toward a future of abundant, diverse, and naturalistic training data for robots. Code and videos are available at https://egozero-robot.github.io.
References (20)
Human-to-Robot Imitation in the Wild
Shikhar Bahl, Abhi Gupta, Deepak Pathak
Reconstructing Hands in 3D with Transformers
G. Pavlakos, Dandan Shan, Ilija Radosavovic et al.
EgoMimic: Scaling Imitation Learning via Egocentric Video
Simar Kareer, Dhruv Patel, Ryan Punamiya et al.
P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies
Mara Levy, Siddhant Haldar, Lerrel Pinto et al.
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Chen Wang, Haochen Shi, Weizhuo Wang et al.
Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation
Siddhant Haldar, Lerrel Pinto
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
D. Damen, Hazel Doughty, G. Farinella et al.
A survey of robot learning from demonstration
B. Argall, S. Chernova, M. Veloso et al.
Dota 2 with Large Scale Deep Reinforcement Learning
Christopher Berner, Greg Brockman, Brooke Chan et al.
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder et al.
Understanding Human Hands in Contact at Internet Scale
Dandan Shan, Jiaqi Geng, Michelle Shu et al.
Zero-Shot Text-to-Image Generation
A. Ramesh, Mikhail Pavlov, Gabriel Goh et al.
Semantic constraints to represent common sense required in household actions for multimodal learning-from-observation robot
K. Ikeuchi, Naoki Wake, Riku Arakawa et al.
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
Yu-Wei Chao, Wei Yang, Yu Xiang et al.
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
A. Mandlekar, Danfei Xu, Josiah Wong et al.
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu et al.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
K. Grauman, Andrew Westbury, Eugene Byrne et al.
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, A. Blattmann, Dominik Lorenz et al.
Embodied Hands : Modeling and Capturing Hands and Bodies Together * * Supplementary Material * *
Javier Romero, Dimitrios Tzionas
Open Source
Mick Watson
Cited By (20)
Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations
HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Human Preference Modeling Using Visual Motion Prediction Improves Robot Skill Learning from Egocentric Human Video
Point Bridge: 3D Representations for Cross Domain Policy Learning
Emergence of Human to Robot Transfer in Vision-Language-Action Models
Mitty: Diffusion-based Human-to-Robot Video Generation
OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer
X-Humanoid: Robotize Human Videos to Generate Humanoid Videos at Scale
SPIDER: Scalable Physics-Informed Dexterous Retargeting
X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
EmbodiSwap for Zero-Shot Robot Imitation Learning
Developing Vision-Language-Action Model from Egocentric Videos
ARMimic: Learning Robotic Manipulation from Passive Human Demonstrations in Augmented Reality
U-ARM : Ultra low-cost general teleoperation interface for robot manipulation
EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos
Scene Text Detection and Recognition "in light of" Challenging Environmental Conditions using Aria Glasses Egocentric Vision Cameras
Object-centric 3D Motion Field for Robot Learning from Human Videos
iTeach: In the Wild Interactive Teaching for Failure-Driven Adaptation of Robot Perception