EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World
EgoVerse unifies large-scale egocentric human demonstration data, enabling cross-embodiment robot learning with 80k episodes, 1965 tasks, and multi-institution collaboration.
Key Findings
Methodology
The study constructs EgoVerse, a collaborative platform aggregating diverse human demonstration data from academic and industrial partners. Data includes 80,000 episodes across 1965 tasks, 240 scenes, with annotations like 3D hand keypoints, camera motion, and task descriptions. Data are managed via EgoDB, supporting continuous ingestion, standardized processing, and multi-modal encoding with transformer-based models for policy learning. Cross-lab experiments validate that increasing data volume and diversity improve transfer performance across multiple robot embodiments, with aligned data being crucial for effective generalization.
Key Results
- Scaling human demonstration data led to an average 15% increase in robot task success rates across multiple tasks and environments, demonstrating the benefit of large, diverse datasets.
- Scene diversity contributed more significantly to environment generalization than demonstrator diversity, which improved robustness to different human morphologies.
- Experiments across three robot platforms showed consistent improvements, confirming the robustness of the standardized protocols and multi-source data fusion in transfer learning.
Significance
This work addresses a core bottleneck in robot learning—data scarcity—by establishing a scalable, collaborative, and standardized human demonstration ecosystem. It enables robots to learn from vast, varied behaviors, reducing reliance on costly robot data collection. The platform's design promotes reproducibility and industrial relevance, accelerating progress toward autonomous robots capable of adapting to complex real-world scenarios. It also bridges the gap between academic research and industrial deployment, fostering a new era of data-driven robot intelligence.
Technical Contribution
The paper introduces EgoVerse, a multi-source, continuously growing dataset with standardized protocols, managed by EgoDB. It employs multi-modal encoders and transformer architectures for policy training, validated through multi-lab, multi-robot experiments. The systematic evaluation demonstrates how data scale and diversity influence transfer performance, providing a blueprint for scalable robot learning frameworks. The integration of industrial and academic data sources exemplifies a novel collaborative approach to large-scale robot training.
Novelty
This is the first large-scale, multi-institutional platform combining continuous human demonstration data collection with systematic cross-embodiment validation. Unlike static datasets, EgoVerse supports ongoing growth and real-world diversity, emphasizing standardization and multi-source fusion. It pioneers comprehensive validation of data scale and diversity effects on robot policy transfer, setting a new benchmark for collaborative, scalable robot learning research.
Limitations
- Dependence on specific hardware setups may introduce biases or data quality issues; hardware variations can affect consistency.
- Current tasks focus on manipulation, limiting generalization to more complex or dynamic scenarios.
- Data privacy and security across institutions pose challenges for open sharing and long-term sustainability.
Future Work
Future directions include expanding task diversity, integrating more advanced multi-modal sensing, and enhancing robustness for real-world deployment. Promoting open access and community contributions will accelerate platform evolution. Exploring self-supervised learning and reinforcement learning techniques with the dataset can further improve autonomous adaptation. Addressing privacy concerns and developing standardized evaluation benchmarks will be key to broader adoption.
AI Executive Summary
Robots have long struggled with learning from limited, static datasets, which restricts their ability to operate effectively in complex, real-world environments. Traditional data collection methods are costly and slow, making it difficult to scale up the diversity and volume needed for robust generalization. Recognizing this bottleneck, the authors introduce EgoVerse—a collaborative, scalable platform that aggregates egocentric human demonstration data from around the world.
EgoVerse leverages diverse hardware setups, including academic-grade Project Aria glasses, industrial custom rigs, and accessible smartphones, to capture rich manipulation behaviors across numerous scenes and tasks. All data are processed into a unified format, including 3D hand poses, camera motion, and task annotations, and stored in EgoDB, a cloud-based system supporting continuous data ingestion and retrieval. This infrastructure enables the platform to grow organically, incorporating contributions from multiple institutions and industries.
A key innovation lies in the systematic validation of how demonstration data scale and diversity influence robot policy transfer. Experiments conducted across three distinct robot platforms—each with different kinematics and sensing configurations—demonstrate that increasing data volume improves success rates by approximately 15%, with scene diversity playing a critical role in environment generalization. These findings are robust, reproducible, and highlight the importance of aligned, diverse data for effective cross-embodiment transfer.
The broader impact of this work is profound. By establishing a standardized, collaborative ecosystem for large-scale human demonstration data, it paves the way for more adaptable, autonomous robots capable of learning from natural human behaviors. This approach reduces the reliance on costly robot data collection, accelerates research, and bridges the gap between academia and industry. Future efforts will focus on expanding task diversity, enhancing multi-modal sensing, and fostering open community participation, ultimately aiming for robots that can seamlessly operate in the unpredictable complexity of real-world settings.
Deep Dive
Abstract
Robot learning increasingly depends on large and diverse data, yet robot data collection remains expensive and difficult to scale. Egocentric human data offer a promising alternative by capturing rich manipulation behavior across everyday environments. However, existing human datasets are often limited in scope, difficult to extend, and fragmented across institutions. We introduce EgoVerse, a collaborative platform for human data-driven robot learning that unifies data collection, processing, and access under a shared framework, enabling contributions from individual researchers, academic labs, and industry partners. The current release includes 1,362 hours (80k episodes) of human demonstrations spanning 1,965 tasks, 240 scenes, and 2,087 unique demonstrators, with standardized formats, manipulation-relevant annotations, and tooling for downstream learning. Beyond the dataset, we conduct a large-scale study of human-to-robot transfer with experiments replicated across multiple labs, tasks, and robot embodiments under shared protocols. We find that policy performance generally improves with increased human data, but that effective scaling depends on alignment between human data and robot learning objectives. Together, the dataset, platform, and study establish a foundation for reproducible progress in human data-driven robot learning. Videos and additional information can be found at https://egoverse.ai/
References (20)
EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Data
Ryan Punamiya, Dhruv Patel, Patcharapong Aphiwetsa et al.
EgoMimic: Scaling Imitation Learning via Egocentric Video
Simar Kareer, Dhruv Patel, Ryan Punamiya et al.
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
Philipp Wu, Yide Shentu, Zhongke Yi et al.
Recognition
Shin-Jye Lee, Ching-Hsun Tseng, T. –. Lin et al.
Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos
Haoyu Xiong, Quanzhou Li, Yun-Chun Chen et al.
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski et al.
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
D. Damen, Hazel Doughty, G. Farinella et al.
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos
Laura M. Smith, Nikita Dhawan, Marvin Zhang et al.
Deep Residual Learning for Image Recognition
Kaiming He, X. Zhang, Shaoqing Ren et al.
Affordances from Human Videos as a Versatile Representation for Robotics
Shikhar Bahl, R. Mendonca, Lili Chen et al.
RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
Haoshu Fang, Hongjie Fang, Zhenyu Tang et al.
Project Aria: A New Tool for Egocentric Multi-Modal AI Research
K. Somasundaram, Jing Dong, Huixuan Tang et al.
Learning Universal Policies via Text-Guided Video Generation
Yilun Du, Mengjiao Yang, Bo Dai et al.
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0
A. Padalkar, A. Pooley, Ajinkya Jain et al.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
K. Grauman, Andrew Westbury, L. Torresani et al.
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
Cheng Chi, Zhenjia Xu, Chuer Pan et al.
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Alexander Khazatsky, Karl Pertsch, S. Nair et al.
Track2Act: Predicting Point Tracks from Internet Videos Enables Generalizable Robot Manipulation
Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta et al.
Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking
Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon et al.
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
Xu-Xin Cheng, Jialong Li, Shiqi Yang et al.
Cited By (20)
EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data
HumanNet: Scaling Human-centric Video Learning to One Million Hours
How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning
EgoKit: Towards Unified Low-Cost Egocentric Data Collection with Heterogeneous Devices
τ0-WM: A Unified Video-Action World Model for Robotic Manipulation
AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning
StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video
CRAFT: A Tendon-Driven Hand with Hybrid Hard-Soft Compliance
HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations
ActiveScale: Scaling Active Perception for Robots across Model, Data, and Hardware
HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
Duet: Dual-Robot Understanding via Efficient Teaching
HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining
PanoVine: Whole-Body Visuomotor Control for Soft Growing Vine Robot
Grounding Generative Policies in Physics: Optimization-Guided Diffusion for Robot Control
Learning Action Priors for Cross-embodiment Robot Manipulation
Not All Actions Are Equal: Rethinking Conditioning for Dexterous World Model
HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation