EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning
EgoInfinity is a modular web-scale 4D hand-object interaction engine that automatically extracts high-quality data from internet videos for robot retargeting and learning.
Key Findings
Methodology
EgoInfinity employs a modular pipeline integrating hand pose estimation, target detection, monocular depth recovery, interaction-aware refinement, and motion retargeting. It leverages algorithms like MOGE-2 for camera calibration, SAM-3 and SAM-3D for object detection and reconstruction, and a cross-module calibration framework to unify scales and coordinate frames. The system uses a neural SE(3)-equivariant network to predict robot-specific root transformations, enabling the conversion of human hand motions into executable robot trajectories. The pipeline operates automatically on large-scale internet videos, without manual annotations, producing high-fidelity 4D hand-object interaction data suitable for cross-embodiment robot learning.
Key Results
- On perception fidelity, EgoInfinity achieves an average hand trajectory error of 2.86cm and surpasses existing methods in gesture recognition accuracy, demonstrating robustness across complex scenes.
- In motion retargeting, the system successfully converts human hand motions into executable trajectories for robots like Unitree G1 and NASA Robonaut2, with IK success rates over 77% and task performance matching task requirements.
- For real robot skill acquisition, the system supports diverse tasks such as grasping, cutting, wiping, and pouring, with high task success rates and natural motion quality, validating the data's practical utility and generalization.
Significance
This work addresses the critical bottleneck of scalable, diverse, and physically grounded manipulation data for robot learning. By automating data extraction from unstructured internet videos, it reduces reliance on costly manual annotations and specialized hardware. The cross-embodiment motion retargeting enables transferability across different robot platforms, accelerating the development of autonomous, adaptable robots capable of operating in open-world environments. It lays a foundation for large-scale, real-world robot training, bridging the gap between human demonstrations and robotic execution.
Technical Contribution
The paper introduces a fully automated, modular data engine combining cross-module scale calibration, interaction-aware trajectory refinement, and a neural SE(3)-equivariant retargeting network. It innovates in unifying diverse perception modules into a consistent physical space, reducing drift and inconsistencies common in visual-only reconstructions. The motion retargeter supports multiple robot morphologies without requiring exact kinematic imitation, enabling flexible, scalable deployment across platforms. These contributions significantly advance the state-of-the-art in automatic manipulation data generation and cross-embodiment transfer.
Novelty
This is the first system capable of automatically generating high-quality 4D hand-object interaction data from web-scale videos without manual labels or specialized sensors. It uniquely combines interaction-aware trajectory calibration with a neural SE(3)-equivariant retargeting network, supporting diverse viewpoints and partial observations. Unlike prior work limited to static datasets or controlled environments, it enables large-scale, in-the-wild data collection and transfer, opening new avenues for scalable robot learning.
Limitations
- The system performs best in near-static camera scenarios; dynamic camera movements or rapid scene changes can introduce errors in reconstruction and calibration.
- Object detection and tracking may struggle under occlusion or cluttered backgrounds, affecting trajectory accuracy.
- The motion retargeting network, while flexible, may face challenges with highly constrained or non-standard robot kinematics, requiring further adaptation.
Future Work
Future research will focus on enhancing robustness in dynamic scenes, integrating multi-view data for improved accuracy, and extending the framework to support multi-robot collaboration. Additionally, incorporating online learning and active perception could further improve data quality and system adaptability in real-world applications.
AI Executive Summary
Internet videos contain a vast repository of human manipulation behaviors, yet transforming this raw visual data into actionable robot training material remains a significant challenge. Traditional datasets, often limited in scale and diversity, cannot fully capture the complexity of open-world manipulation tasks. To address this, the authors introduce EgoInfinity, a modular, fully automated 4D hand-object interaction data engine capable of processing web-scale videos.
EgoInfinity integrates advanced perception modules—including hand pose estimation, object detection, monocular depth recovery, and interaction classification—within a unified calibration framework. This ensures consistent, physically plausible reconstructions by aligning scales and coordinate frames across modules. The system employs a neural SE(3)-equivariant network to predict robot-specific root transformations, enabling the conversion of human hand motions into robot-compatible joint trajectories. This approach allows for flexible, cross-embodiment retargeting, even from partial observations or arbitrary viewpoints.
Experimental validation demonstrates the system’s ability to produce high-fidelity 4D data, with average hand trajectory errors below 3cm and over 77% IK success rates across multiple robots. The generated data supports diverse tasks such as grasping, pouring, and wiping, which are successfully transferred to real robots, including Unitree G1, NASA Robonaut2, and Franka FR3. These results showcase the potential of EgoInfinity to significantly accelerate robot learning by leveraging massive, unstructured internet videos.
While promising, the system faces limitations in dynamic scenes, occlusion scenarios, and highly constrained robot kinematics. Future work aims to improve robustness, incorporate multi-view data, and extend multi-robot capabilities. Overall, EgoInfinity offers a scalable, versatile platform for turning the wealth of online human videos into practical, large-scale robot training datasets, paving the way for more autonomous and adaptable robotic systems in complex environments.
Deep Dive
Abstract
Internet videos constitute the largest reservoir of embodied human manipulation knowledge, yet converting arbitrary RGB footage into actionable robot training data remains a major bottleneck. Existing lab- or factory-collected datasets are narrow in scale and diversity, limiting open-world robot learning. Instead of proposing a static dataset, we introduce EgoInfinity, a universal 4D hand-object interaction data engine that enables web-scale data generation for robot retargeting and learning. EgoInfinity is a modular engine integrating perception, segmentation, reconstruction, interaction-aware refinement, and retargeting to automate this traditionally unscalable video-to-action problem without human-in-the-loop annotation. Its modular design lets the engine continuously benefit from advances in any incorporated component. With EgoInfinity, in-the-wild human manipulation videos are lifted into agent-agnostic, metric 4D hand-object representations, including hand trajectories, 6-DoF object poses, and contact-relevant states. Rather than naively connecting standalone components, EgoInfinity combines cross-module metric calibration with interaction-aware refinement to improve physical reliability, reducing drift and contact inconsistencies common in pure visual reconstruction. We further propose a novel motion retargeter that compiles the recovered 3D hand motions into executable joint trajectories for diverse robot morphologies, enabling video-to-action retargeting on any robot from arbitrary viewpoints and shot sizes (e.g., the human body is only partially visible). We validate EgoInfinity across perception fidelity, kinematic feasibility, contact consistency, cross-embodiment generalization, and real-robot skill acquisition (e.g., grasping, cutting, wiping, and pouring), demonstrating a scalable bridge from internet videos to executable robot behavior for open-world robot learning.