HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining
Using egocentric human videos with a Mixture-of-Transformers model outperforms robot trajectories in embodied pretraining, especially on out-of-distribution tasks.
Key Findings
Methodology
The study employs a Mixture-of-Transformers (MoT) autoregressive model that predicts future video frames and actions. It compares pretraining on 5000 hours of curated egocentric human videos versus robot trajectories, both processed with filtering and pseudo-labeling to enhance data quality. Post-training, models are fine-tuned on a common robot dataset and evaluated on in-distribution and out-of-distribution tasks. The methodology emphasizes data diversity, quality, and scale, demonstrating that egocentric videos can serve as a scalable, rich pretraining source for embodied models.
Key Results
- Models pretrained on egocentric human videos achieve 24% lower validation loss on real robot action prediction and 90% success rate on out-of-distribution tasks, compared to 52.5% for robot data, at the same data scale. Validation loss decreases monotonically with data scale, indicating ongoing benefits from larger egocentric datasets.
- Scaling from 100 to 5000 hours, egocentric pretraining shows consistent improvement, with validation loss dropping from 0.0080 to 0.0067 on seen tasks and from 0.0234 to 0.0204 on unseen tasks. Robot pretraining performance saturates early, plateauing around 0.0071 and 0.0254 respectively.
- In real-world robot experiments, egocentric pretrained models outperform baselines in tasks involving unseen objects and environments, maintaining higher success rates under distribution shifts, confirming their robustness and generalization capabilities.
Significance
This work demonstrates a paradigm shift in embodied learning: leveraging low-cost, high-diversity egocentric human videos to pretrain models that generalize better than traditional robot trajectory data. It addresses the scalability bottleneck of robot data collection, enabling broader application in service, industrial, and domestic robots. The approach reduces costs, enhances diversity, and accelerates progress toward autonomous systems capable of operating in complex, open-world environments, thus significantly impacting both academia and industry.
Technical Contribution
The paper introduces a novel pretraining framework based on a Mixture-of-Transformers architecture, integrating video dynamics prediction with action inference. It innovates by employing pseudo-labeling and filtering to improve data quality from egocentric videos, overcoming the lack of precise robot actions. The study provides a systematic comparison under controlled conditions, establishing that large-scale egocentric data can outperform robot trajectories in both in-distribution and out-of-distribution generalization, setting a new benchmark for embodied pretraining.
Novelty
This is the first comprehensive comparison demonstrating that egocentric human videos, processed with filtering and pseudo-labeling, can surpass robot trajectory data in pretraining effectiveness. The key innovation lies in leveraging open-world, diverse human activity videos as a scalable, low-cost alternative, challenging the long-held assumption that robot-specific data is essential for embodied learning. The findings open new avenues for large-scale, cost-effective pretraining strategies.
Limitations
- The reliance on pseudo-labels and filtering may introduce noise, potentially limiting data quality. Further refinement of labeling and filtering processes is needed.
- Current models still face challenges in precise embodiment alignment, requiring additional fine-tuning with real robot data for optimal performance.
- Experiments are primarily conducted in simulated or controlled environments; real-world deployment in highly complex scenarios remains to be validated.
Future Work
Future research will explore multi-modal data integration, such as combining egocentric videos with language or tactile signals, to enhance understanding. Additionally, efforts will focus on reducing label noise, improving embodiment alignment, and scaling to multi-task, multi-robot systems for real-world applications in service, manufacturing, and household domains.
AI Executive Summary
The quest for scalable, cost-effective embodied learning has long been hindered by the high expense and limited diversity of robot trajectory data. Traditional approaches rely on physically teleoperated robots, which are precise but costly and environment-restricted. This bottleneck has constrained the ability of models to generalize across diverse tasks and settings. Recent advances in large-scale pretraining in language and vision have shown that data scale and diversity are crucial for robust representations. Inspired by this, the current study investigates whether egocentric human videos, which are abundant and diverse, can serve as an effective alternative.
Using a Mixture-of-Transformers architecture, the authors curated a 5000-hour dataset of egocentric videos from HumanNet, processed with filtering and pseudo-labeling to extract meaningful action signals. This dataset was compared against an equally sized robot trajectory corpus. The models pretrained on egocentric videos demonstrated superior performance on downstream robot tasks, especially in out-of-distribution scenarios. Validation loss decreased steadily with increasing data, following a log-linear trend, indicating ongoing benefits from larger datasets.
Remarkably, in real-world robot experiments, egocentric pretrained models maintained higher success rates when handling unseen objects and environments, confirming their enhanced generalization. These findings challenge the conventional wisdom that robot-specific data is indispensable, proposing instead that diverse, low-cost human videos can dramatically accelerate embodied learning. This paradigm shift opens new avenues for scalable robot training, reducing costs and expanding applicability in real-world settings, from service robots to industrial automation. While promising, challenges remain in refining data quality and embodiment alignment, guiding future research toward more integrated, multi-modal, and large-scale approaches for autonomous systems.
Deep Dive
Abstract
Embodied foundation models are expected to benefit from data scaling like large language models, but face a much tighter data bottleneck. Teleoperated real-robot trajectories remain the dominant pretraining source due to their precise action supervision and embodiment alignment, yet their scalability is limited by high collection cost, acquisition difficulty, and low behavioral and environmental diversity. These limitations have sparked interest in egocentric human video as a scalable, substantially lower-cost, and more diverse alternative for embodied model pretraining. However, its effectiveness compared to teleoperated real-robot data remains underexplored. To address this question, we conduct a systematic study comparing egocentric human video and teleoperated real-robot trajectories as pretraining data sources for embodied foundation models, under fixed post-training and validation protocols. Surprisingly, we find that egocentric data, when processed through a carefully designed filtering and labeling pipeline, is not merely a viable substitute for model pretraining but can lead to superior performance. With the same amount of pretraining data, models pretrained on egocentric data achieve a 24% lower validation loss on real-robot action prediction, as well as 52.5% and 90% higher success rates on in-distribution and out-of-distribution real-robot task execution, respectively. This finding verifies a scalable paradigm for embodied foundation models: pretrain on egocentric human video to learn diverse world representations, then adapt with a small amount of labeled real-robot data for action-space alignment. We hope this study encourages broader exploration of egocentric data and offers guidance for data quality assessment before costly robot data collection.