ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
ConLA employs contrastive disentanglement to learn pure latent actions from human videos, surpassing robot trajectory pretraining with only video data.
Key Findings
Methodology
ConLA introduces a contrastive disentanglement framework combining action category priors and temporal cues. Using VQ-VAE, it extracts motion features from frame pairs, then applies contrastive loss to separate action from visual noise. The architecture includes a spatial-temporal Transformer encoder, action and visual contrastive heads, and a quantization module. The first stage produces discrete latent action tokens; the second stage trains a vision-language model to predict these tokens from observations and instructions; the third fine-tunes the policy on robot data. This approach effectively mitigates shortcut learning, ensuring semantically meaningful latent representations suitable for transfer.
Key Results
- On the SimplerEnv benchmark, ConLA pretrained solely on human videos achieved a success rate increase of 12.5%, outperforming models pretrained on real robot trajectories. In real robot experiments, success rates exceeded baseline by 10%, demonstrating strong transferability.
- Compared to LAPA and SCRATCH, ConLA showed significant improvements in multi-task success, with over 10% gains. Ablation studies confirmed that the contrastive disentanglement mechanism enhances semantic purity and transfer performance.
- The learned latent actions exhibit high semantic consistency, enabling robust transfer across diverse environments and tasks, even with noisy human videos, validating the effectiveness of the proposed contrastive approach.
Significance
This work addresses the challenge of leveraging large-scale human videos for robotic policy learning, reducing reliance on costly robot data. By disentangling motion semantics from visual distractions, ConLA unlocks the potential of internet-scale videos, enabling scalable, cost-effective robot training. It bridges the gap between human demonstration richness and robot learning requirements, paving the way for more autonomous, adaptable robots in industry and service sectors. The approach also advances unsupervised representation learning, with broad implications for AI generalization and transferability.
Technical Contribution
ConLA's core innovation lies in the contrastive disentanglement architecture that combines action category priors and temporal cues to produce semantically pure latent actions. It integrates VQ-VAE for discrete action quantization with a contrastive loss to prevent shortcut learning. The architecture employs a spatial-temporal Transformer encoder and dual contrastive heads, enabling effective separation of motion and appearance features. This design surpasses traditional reconstruction-based methods, providing more robust, transferable representations suitable for downstream robotic control, even without explicit action labels.
Novelty
This is the first work to incorporate contrastive disentanglement with action category and temporal priors for latent action learning from human videos in robotics. Unlike prior approaches relying solely on visual reconstruction or unsupervised inverse dynamics, ConLA explicitly enforces semantic clustering and motion-content separation. Its ability to outperform robot trajectory pretraining solely with human videos demonstrates a significant leap in leveraging unstructured, large-scale internet videos for robotic policy transfer, marking a new paradigm in scalable robot learning.
Limitations
- The model's performance may degrade in scenarios with highly ambiguous or complex actions lacking clear category labels or with severe visual noise, limiting generalization in extreme conditions.
- The computational overhead of Transformer-based encoders and contrastive mechanisms may hinder real-time deployment, requiring further optimization.
- Dependence on accurate category labels and temporal cues means that noisy annotations or disrupted temporal sequences could impair disentanglement quality.
Future Work
Future directions include integrating multimodal signals such as audio or tactile feedback to enrich action representations, exploring reinforcement learning to improve policy robustness, and reducing computational costs for real-time applications. Extending to multi-agent and multi-task settings, as well as applying to more complex manipulation tasks, will further demonstrate the scalability and versatility of ConLA.
AI Executive Summary
The rapid development of autonomous robots hinges on efficient learning from diverse data sources. Traditional methods rely heavily on costly robot trajectory datasets, limiting scalability. Human demonstration videos, abundant and diverse, offer a promising alternative but pose challenges due to lack of explicit action labels and visual noise. ConLA addresses this by introducing a contrastive disentanglement framework that leverages action category priors and temporal cues to extract pure, semantically meaningful latent actions from human videos. Using a combination of VQ-VAE, spatial-temporal Transformer encoders, and contrastive learning, the method effectively separates motion dynamics from visual distractions, mitigating shortcut learning issues common in prior approaches.
Experimental results demonstrate that ConLA, pretrained solely on human videos, surpasses models trained on real robot trajectories in benchmark tasks like SimplerEnv, achieving a 12.5% success rate improvement. In real-world robotic experiments, it outperforms baseline methods by over 10%, validating its transferability and robustness. These findings highlight the potential of large-scale human videos to revolutionize robot learning, drastically reducing data collection costs while maintaining high performance.
This work not only advances the state-of-the-art in latent action learning but also opens new avenues for scalable, cost-effective robot training. By effectively disentangling action semantics from visual noise, ConLA paves the way for autonomous systems capable of learning complex manipulation skills from internet-scale data. Future research will focus on multimodal integration, real-time deployment, and extending to multi-robot systems, aiming to realize truly scalable autonomous agents capable of operating in unstructured environments.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里学做菜，你的视频里有很多不同的动作，比如切菜、炒菜，但背景有碗碟、油渍，非常杂乱。传统的方法就像是让你模仿每个细节，容易记错或被背景干扰。ConLA的方法像是用一种聪明的眼镜，只看动作的核心，比如“切菜”或“炒菜”，把背景的杂乱都过滤掉。它还会根据动作的类别，把相似的动作放在一起，让学习变得更快更准。这样，机器人也能从人类的视频中学会做菜，不再被背景迷惑，变得更聪明、更会干活。
Abstract
Vision-Language-Action (VLA) models achieve preliminary generalization through pretraining on large scale robot teleoperation datasets. However, acquiring datasets that comprehensively cover diverse tasks and environments is extremely costly and difficult to scale. In contrast, human demonstration videos offer a rich and scalable source of diverse scenes and manipulation behaviors, yet their lack of explicit action supervision hinders direct utilization. Prior work leverages VQ-VAE based frameworks to learn latent actions from human videos in an unsupervised manner. Nevertheless, since the training objective primarily focuses on reconstructing visual appearances rather than capturing inter-frame dynamics, the learned representations tend to rely on spurious visual cues, leading to shortcut learning and entangled latent representations that hinder transferability. To address this, we propose ConLA, an unsupervised pretraining framework for learning robotic policies from human videos. ConLA introduces a contrastive disentanglement mechanism that leverages action category priors and temporal cues to isolate motion dynamics from visual content, effectively mitigating shortcut learning. Extensive experiments show that ConLA achieves strong performance across diverse benchmarks. Notably, by pretraining solely on human videos, our method for the first time surpasses the performance obtained with real robot trajectory pretraining, highlighting its ability to extract pure and semantically consistent latent action representations for scalable robot learning.