ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
ACE-EGO-0 unifies human and robot data via spatial, structural, and temporal alignment for VLA pretraining, boosting manipulation performance.
Key Findings
Methodology
ACE-EGO-0 introduces a unified action representation through spatial (camera coordinates), structural (morphology conditioning), and temporal (action chunking) alignment, integrating diverse robot and human videos. Pseudo-action pipelines convert raw human videos into robot-compatible trajectories, with a reliability-aware loss to mitigate label noise. The model leverages 6K+ hours of multi-source data, combining real robot, simulation, and pseudo-labeled human videos, effectively addressing heterogeneity and noise issues.
Key Results
- Achieved 72.8% success on RoboCasa GR1 tabletop tasks, outperforming baselines by over 10%. On RoboTwin 2.0, success rates reached 91.12% (Easy) and 90.62% (Hard). Demonstrated strong transfer to real-world bimanual tasks, outperforming prior methods.
- Incorporating large-scale human pseudo-labels improved generalization across tasks and environments. Reliability-aware weighting reduced noise impact, stabilizing training. Ablation studies confirmed the importance of each alignment component.
- The approach significantly advances multi-source data integration, enabling scalable pretraining that enhances downstream manipulation capabilities in complex scenarios.
Significance
This work addresses the longstanding challenge of integrating heterogeneous embodied data for robot learning. By systematically aligning multi-modal, multi-embodiment data and controlling label noise, it paves the way for scalable, generalizable robotic systems. The framework enhances the ability of robots to learn from diverse experiences, bridging the gap between virtual demonstrations, human videos, and real-world tasks, thus accelerating progress toward autonomous, adaptable robots in industry and daily life.
Technical Contribution
The core innovation lies in the spatial-structural-temporal alignment framework combined with a reliability-aware loss, enabling effective multi-source data fusion. The unified camera-space action representation simplifies cross-embodiment learning, while morphology conditioning and time-aligned chunking improve robustness. This approach extends the state-of-the-art in scalable, multimodal robot pretraining, offering new avenues for research and application.
Novelty
This is the first comprehensive system to unify large-scale human pseudo-action videos with robot data via multi-dimensional alignment, addressing both representation heterogeneity and label noise simultaneously. Unlike prior works focusing on single-source or simple fusion, ACE-EGO-0 offers a principled, scalable solution that significantly outperforms existing methods in manipulation benchmarks.
Limitations
- The pseudo-labels depend heavily on hand reconstruction and pose estimation accuracy, which can be compromised by occlusion or complex backgrounds, limiting label fidelity.
- While effective in many scenarios, the model's performance in extremely long-horizon or highly dynamic environments remains to be validated, requiring further robustness improvements.
- Training on large multi-source datasets incurs high computational costs, and optimizing efficiency is an ongoing challenge for practical deployment.
Future Work
Future directions include enhancing pseudo-label quality via self-supervised refinement, integrating additional modalities such as speech and tactile feedback, and developing more efficient training algorithms. Extending the framework to broader real-world applications, including autonomous service robots and industrial automation, will be key. Additionally, exploring lifelong learning paradigms to continually improve from ongoing multi-source data streams is a promising avenue.
AI Executive Summary
Robotics research has long sought to develop systems capable of versatile, autonomous manipulation across diverse environments. Traditional approaches relied heavily on costly, task-specific demonstrations, limiting scalability and generalization. Recent advances in large-scale pretraining, inspired by language and vision models, have shown promise but face significant hurdles due to the heterogeneity of data sources—ranging from robot logs to human videos.
ACE-EGO-0 introduces a novel framework that unifies these disparate data streams through spatial, structural, and temporal alignment. By projecting human and robot actions into a shared camera-centric coordinate system, conditioning on morphology, and segmenting actions by physical duration, the model effectively bridges the gap between different embodiments and control frequencies. A key innovation is the pseudo-action pipeline, which converts raw egocentric videos into robot-compatible trajectories, enabling large-scale, low-cost supervision.
To handle the inherent noise in human pseudo-labels, ACE-EGO-0 employs a reliability-aware training objective. This mechanism dynamically weights supervision signals based on their estimated fidelity, ensuring that high-quality robot data anchors the learning process while noisy human labels serve as auxiliary guidance. The model is trained on over 6,000 hours of multi-source data, including real robot demonstrations, simulations, and pseudo-labeled human videos.
Experimental results demonstrate state-of-the-art performance on benchmarks such as RoboCasa and RoboTwin 2.0, with success rates surpassing previous methods by significant margins. The framework also exhibits strong transfer to real-world bimanual manipulation tasks, validating its generalization capability. Ablation studies confirm the importance of each alignment component and the reliability mechanism.
This work marks a substantial step toward scalable, general-purpose robotic learning, offering a practical pathway to leverage vast amounts of human video data alongside traditional robot demonstrations. Future efforts will focus on improving pseudo-label accuracy, expanding multimodal inputs, and deploying in more complex, real-world scenarios, ultimately bringing autonomous, adaptable robots closer to everyday life.
Deep Analysis
Background
机器人自主学习经历了从模仿特定任务到追求通用智能的演变。RT-1、RT-2、OpenVLA等代表性工作推动了大规模数据驱动的预训练策略，但数据异质性和标签噪声成为限制模型泛化的瓶颈。人类视频作为廉价丰富的补充资源逐渐受到关注，尤其在长尾行为和复杂场景中表现优异。然而，伪标签的噪声和多源融合的技术难题仍未完全解决，限制了其在实际应用中的效果。
Core Problem
核心问题在于如何高效融合异质的机器人和人类数据，克服空间、结构和时间上的差异，以及伪标签的噪声干扰。传统方法多依赖单一数据源或简单融合策略，难以实现跨源信息的充分利用。这不仅影响模型的泛化能力，也限制了机器人在复杂环境中的自主学习潜力。解决这一问题对于推动机器人向更高层次的自主性和适应性发展具有重要意义。
Innovation
本研究提出空间-结构-时间三维对齐的统一动作表示，解决多源数据的异质性。引入伪动作生成管线，结合可靠性感知机制，有效过滤噪声，提升标签质量。模型采用多源融合策略，结合形态条件和时间对齐，增强模型的泛化能力。创新点在于系统性整合大规模人类伪动作和机器人数据，突破了现有技术的局限。
Methodology
- �� 构建空间对齐：将机器人和人类动作投影到相机坐标系，避免复杂空间变换。
- �� 结构对齐：引入形态条件，将不同机器人和人类手部结构映射到共享的形态空间。
- �� 时间对齐：采用物理时间尺度定义动作块，确保不同控制频率的数据在时间上的一致性。
- �� 伪标签生成：利用手部重建和动作估计，将人类视频转化为伪动作轨迹。
- �� 可靠性感知：根据标签噪声和追踪稳定性动态调整伪标签的权重。
- �� 多源融合：结合真实机器人、模拟和伪动作数据，进行联合预训练。
Experiments
采用6千小时多源数据，包括机器人示范、模拟和大规模人类视频伪标签。评估在RoboCasa和RoboTwin 2.0等基准上，比较不同对齐机制和噪声控制策略的效果。指标包括成功率、迁移性能和泛化能力，采用标准的任务成功率和平均成功率作为评价指标。消融实验验证空间、结构和时间对齐的重要性，模型在复杂任务中的表现优于对比方法。
Results
模型在RoboCasa GR1桌面任务中成功率达72.8%，优于基线10%以上。在RoboTwin 2.0的Easy和Hard分割中，成功率分别达到91.12%和90.62%。在真实双手操控任务中表现出优异的迁移能力，验证了多源融合和伪标签机制的有效性。消融实验显示空间、结构和时间对齐机制对性能提升贡献显著，伪标签质量控制提升训练稳定性。
Applications
该方法适用于机器人自主学习、复杂环境操控、多任务迁移等场景。通过大规模伪标签和多源数据融合，提升机器人在实际环境中的适应性和自主性。未来可结合自然语言指令和多模态信息，推动机器人实现更智能的自主操作，广泛应用于工业、家庭和服务机器人领域。
Limitations & Outlook
伪标签生成依赖于手部重建和动作估计的准确性，可能在遮挡或复杂背景下出现偏差，影响标签质量。模型在极端长时序和复杂环境中的表现仍有限，需增强鲁棒性。训练成本较高，未来需优化数据处理和模型效率，提升实际应用的可行性。
Plain Language Accessible to non-experts
想象你在厨房准备一顿大餐。你有不同的厨具、不同的食材、不同的菜谱，有些是你熟悉的，有些是新学的。为了做出美味的菜肴，你需要把这些不同的工具和食材放在一个统一的操作平台上，按照一定的顺序和时间安排。有人帮你把复杂的动作拆分成简单的步骤，有的步骤你很确定，有的步骤可能有点不准。这就像ACE-EGO-0，把各种不同的厨房工具和菜谱整理成一个统一的流程，确保每个步骤都能顺利完成，最终做出美味佳肴。
ELI14 Explained like you're 14
想象你在学校的科学实验室做实验。有时候用老师的设备，有时候用自己带的不同工具。为了让实验顺利，你得把这些不同的设备和步骤整理成一个统一的流程。比如，老师的设备用一种方式操作，你的朋友用另一种，但都要达到相同的实验目的。这就像ACE-EGO-0，把不同的机器人和人类操作变成一样的语言，让它们都能理解合作。这样，机器人就能像人一样学会做很多复杂的事情，无论是在实验室还是在家里，都能帮你完成任务。
Abstract
Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.