Developing Vision-Language-Action Model from Egocentric Videos
Using EgoScaler to automatically extract 6DoF object trajectories from unlabeled egocentric videos significantly improves VLA pre-training, achieving over 20% success rate gains.
Key Findings
Methodology
This paper introduces a framework leveraging EgoScaler to automatically extract object manipulation trajectories from four large egocentric video datasets without auxiliary labels. The process involves identifying action start/end points via GPT-4o, extracting object positions with open-vocabulary segmentation and dense point tracking, projecting trajectories into camera coordinates through point cloud registration, and computing rotations via SVD. The resulting 6DoF trajectories are filtered to remove noise and incomplete data, forming a large-scale pretraining dataset. The π0 architecture is trained on this data, evaluated in simulation and real robot environments, demonstrating over 20% improvement in task success rates compared to scratch training, with performance comparable to models trained on real robot datasets. Combining datasets yields further gains, validating the approach’s scalability and effectiveness.
Key Results
- Pre-training on the constructed dataset improves task success rates by over 20%, outperforming scratch models and matching performance of models trained on real robot data, with further improvements when combined.
- The automatic trajectory extraction method maintains high data quality, enabling robust model training across diverse environments and tasks.
- The approach reduces reliance on costly manual annotations, demonstrating a scalable, low-cost pathway for large-scale robot pretraining.
Significance
This work addresses a core bottleneck in robotic foundation model development—data scarcity—by leveraging ubiquitous egocentric videos. It offers a scalable, annotation-free method to generate high-quality training data, bridging the gap between human demonstrations and robot learning. The approach democratizes data collection, enabling broader research and application in autonomous manipulation, service robots, and industrial automation. It also paves the way for multimodal learning paradigms that integrate vision, language, and action, fostering more adaptable and intelligent robotic systems.
Technical Contribution
The paper proposes an innovative, fully automated pipeline for extracting precise 6DoF object trajectories from unlabeled egocentric videos, combining GPT-4o for action segmentation, open-vocabulary segmentation, dense point tracking, point cloud registration, and SVD-based rotation computation. It introduces robust filtering strategies to ensure data quality, enabling large-scale dataset construction. The integration of these components within the EgoScaler framework significantly advances the state-of-the-art in unsupervised trajectory extraction, facilitating scalable pretraining of vision-language-action models. The experimental validation demonstrates that models trained on this data outperform or match those trained on costly real robot datasets, highlighting the method’s practical impact.
Novelty
This is the first work to fully automate the extraction of high-fidelity object manipulation trajectories from unlabeled egocentric videos, eliminating the need for dense manual annotations or specialized hardware. It combines multiple advanced computer vision techniques into a unified pipeline, enabling large-scale, diverse data collection for robotic pretraining. This approach fundamentally shifts the paradigm from labor-intensive data collection to scalable, automated dataset generation, opening new avenues for research in robot learning and multimodal perception.
Limitations
- Trajectory extraction accuracy diminishes in scenes with heavy occlusion, clutter, or complex backgrounds, leading to potential noise and reduced model performance.
- The current filtering strategies may discard some valid trajectories, limiting data diversity and coverage.
- Generalization to highly unstructured or extreme environments remains untested, requiring further robustness improvements.
- Computational costs of processing large datasets and refining trajectories could be significant, necessitating optimization for real-time or resource-constrained applications.
Future Work
Future research will focus on integrating additional modalities such as depth and hand pose data to improve trajectory accuracy, especially in challenging scenarios. Developing adaptive filtering techniques and semi-supervised learning methods could further enhance data quality. Extending the framework to more complex tasks, including multi-object manipulation and dynamic environments, will broaden its applicability. Additionally, efforts to optimize computational efficiency will facilitate deployment in real-time robotic systems, accelerating the transition from research to practical applications.
AI Executive Summary
This study presents a novel approach to overcoming the data bottleneck in robot learning by leveraging egocentric videos without manual annotations. Using the EgoScaler framework, the authors automatically extract high-fidelity object manipulation trajectories from four large-scale egocentric video datasets, including Ego4D and HD-EPIC. The process involves identifying action segments with GPT-4o, segmenting objects with open-vocabulary models, tracking dense point clouds, and computing rotations via SVD, resulting in a comprehensive 6DoF trajectory dataset of over 45,000 episodes. These trajectories are automatically filtered to remove noise, ensuring high data quality.
The authors train a state-of-the-art π0 vision-language-action model on this dataset, evaluating in both simulated and real robot environments. Results show a success rate increase of over 20% compared to models trained from scratch, with performance comparable to models trained on expensive real robot datasets such as BC-Z and BridgeData V2. Combining their dataset with existing robot datasets yields further improvements, demonstrating the scalability and robustness of their approach.
This work signifies a major step toward scalable, low-cost robot pretraining, addressing the critical challenge of data scarcity. By harnessing the ubiquity of egocentric videos, the method democratizes access to large-scale training data, enabling broader research and application in autonomous manipulation. The approach also opens new avenues for multimodal learning, integrating vision, language, and action. Future directions include enhancing trajectory accuracy with additional modalities, extending to more complex tasks, and optimizing computational efficiency, paving the way for more adaptable and intelligent robotic systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房帮忙做饭，你看着视频学习怎么切菜、炒菜。以前，要教机器人学这些动作，得用特殊的相机和手工标记每个动作，既麻烦又贵。现在，科学家用普通的手机视频，让机器人自己看视频，学会了这些动作。它通过分析视频中物体的移动，知道什么时候拿起菜刀，什么时候放到锅里。这就像你看YouTube学做菜一样，机器人也能自己学会很多技能，不用花很多钱，也不用人工标记。这样，机器人可以更快、更便宜地学会新技能，帮我们做家务、工作，变得更聪明、更贴心。
ELI14 Explained like you're 14
想象你在家帮妈妈做饭，你看着视频学习怎么切菜、炒菜。以前，要教机器人学这些动作，得用特殊的相机和手工标记每个动作，既麻烦又贵。现在，科学家用普通的手机视频，让机器人自己看视频，学会了这些动作。就像你看YouTube学做菜一样，机器人通过分析视频中的物体移动，知道什么时候拿起菜刀，什么时候放到锅里。这种方法不用花很多钱，也不用人工标记，就能让机器人学会很多技能。未来，机器人可以更快学会新东西，帮我们做家务、工作，变得更聪明、更贴心。
Abstract
Egocentric videos capture how humans manipulate objects and tools, providing diverse motion cues for learning object manipulation. Unlike the costly, expert-driven manual teleoperation commonly used in training Vision-Language-Action models (VLAs), egocentric videos offer a scalable alternative. However, prior studies that leverage such videos for training robot policies typically rely on auxiliary annotations, such as detailed hand-pose recordings. Consequently, it remains unclear whether VLAs can be trained directly from raw egocentric videos. In this work, we address this challenge by leveraging EgoScaler, a framework that extracts 6DoF object manipulation trajectories from egocentric videos without requiring auxiliary recordings. We apply EgoScaler to four large-scale egocentric video datasets and automatically refine noisy or incomplete trajectories, thereby constructing a new large-scale dataset for VLA pre-training. Our experiments with a state-of-the-art $π_0$ architecture in both simulated and real-robot environments yield three key findings: (i) pre-training on our dataset improves task success rates by over 20\% compared to training from scratch, (ii) the performance is competitive with that achieved using real-robot datasets, and (iii) combining our dataset with real-robot data yields further improvements. These results demonstrate that egocentric videos constitute a promising and scalable resource for advancing VLA research.