R+X: Retrieval and Execution from Everyday Human Videos
R+X leverages vision-language models for retrieval and in-context imitation learning, enabling robots to learn from unlabelled human videos without training.
Key Findings
Methodology
R+X employs a two-stage pipeline: retrieval and execution. In the retrieval phase, a state-of-the-art vision-language model (Gemini Pro 1.5 Flash) is used to extract relevant short video clips from long, unlabelled human videos based on natural language commands. The process involves converting video frames into sparse 3D keypoints using DINO features, and extracting hand trajectories with HaMeR. In the execution phase, a few-shot in-context imitation model (KAT) takes these retrieved examples to generate robot action trajectories conditioned on current observations. This approach fully exploits pre-trained models’ capabilities without any fine-tuning, enabling immediate task execution.
Key Results
- Experiments on 12 household tasks demonstrated that R+X achieved an average success rate of 85%, outperforming baseline models such as finetuned R3M-DiffLang and Octo by 10-20%. The framework showed strong generalization to unseen objects, varied environments, and complex language commands. In continuous learning scenarios, R+X maintained performance with less than 5% degradation, whereas baseline models suffered over 20% performance loss due to catastrophic forgetting. Ablation studies confirmed that the retrieval quality and feature extraction steps critically influence overall performance.
- In spatial and language generalization tests, R+X successfully executed tasks with objects in novel positions and with complex instructions like 'call my mom' or 'turn off the lights,' demonstrating robust understanding and adaptation. The method also proved resilient to distractors, focusing only on semantically relevant keypoints, which enhanced robustness in cluttered scenes. The success rates across various scenarios consistently exceeded 80%, validating the approach’s versatility.
- The ablation experiments highlighted that the combination of DINO-based keypoints, HaMeR hand pose estimation, and Gemini’s retrieval capability are essential for high performance. Omitting any component led to significant drops, emphasizing the importance of integrated multi-modal features for behavior understanding and imitation.
Significance
This work addresses a fundamental challenge in robotics: learning complex behaviors from natural, unstructured data without costly annotations or extensive retraining. By leveraging large-scale pre-trained vision-language models, R+X offers a scalable and flexible solution that bridges the gap between passive observation and active manipulation. It paves the way for robots to autonomously acquire skills in diverse real-world environments, significantly reducing development costs and increasing adaptability. The approach aligns with the broader trend of utilizing foundation models for embodied AI, promising a future where robots can learn continuously from everyday human activities captured via wearable devices. Such capabilities could revolutionize domestic service, industrial automation, and assistive robotics, making intelligent machines more accessible and versatile.
Technical Contribution
The paper introduces a novel framework combining state-of-the-art vision-language retrieval models with few-shot in-context imitation learning. It uniquely employs Gemini Pro 1.5 Flash for behavior retrieval, bypassing the need for manual annotations or task-specific fine-tuning. The use of DINO features for scene keypoints and HaMeR for hand pose estimation ensures rich, semantically meaningful behavior representations. The core innovation is the integration of these components with KAT, enabling zero-shot behavior imitation conditioned on retrieved examples. This modular design enhances generalization, reduces training costs, and facilitates continual learning, setting a new standard for scalable robot skill acquisition.
Novelty
This research is the first to demonstrate that large pre-trained vision-language models can be directly used for behavior retrieval in robotic applications, enabling immediate imitation without fine-tuning. Unlike prior works relying on supervised learning or domain-specific datasets, R+X leverages foundation models’ zero-shot capabilities, combining retrieval with few-shot imitation to handle diverse, unstructured human videos. The approach effectively decouples perception and control, allowing flexible, scalable behavior learning from natural videos, which is a significant departure from traditional end-to-end training paradigms.
Limitations
- The framework heavily depends on the quality and robustness of the pre-trained models like Gemini and DINO; poor performance of these models in certain scenarios can limit retrieval accuracy and behavior understanding.
- In highly dynamic or cluttered environments, keypoint detection and hand pose estimation may introduce errors, affecting imitation fidelity.
- Current validation is limited to single-arm robots and a predefined set of tasks; extending to multi-arm robots, more complex tasks, or outdoor environments remains an open challenge.
Future Work
Future directions include enhancing retrieval accuracy with multi-modal cues such as audio or tactile signals, enabling more nuanced behavior understanding. Developing online learning capabilities for robots to adapt continuously from new videos, and scaling the approach to multi-robot systems and more complex tasks are also promising avenues. Additionally, integrating reinforcement learning with retrieval-based imitation could further improve robustness and autonomy in unstructured real-world settings.
AI Executive Summary
The quest for autonomous robots capable of learning complex behaviors from natural human activities has long been hindered by the reliance on labor-intensive data annotation and task-specific training. Traditional methods, such as behavior cloning and reinforcement learning, require extensive labeled datasets and often lack the flexibility to generalize across diverse environments. Recent advances in foundation models, particularly vision-language models like Gemini Pro 1.5 Flash, have opened new horizons by enabling zero-shot understanding and retrieval of visual behaviors. Building upon this, the R+X framework presents a paradigm shift in robot skill acquisition.
At its core, R+X is a two-stage pipeline that leverages pre-trained models for behavior retrieval and imitation. The retrieval stage employs Gemini Pro 1.5 Flash to scan long, unlabelled videos of humans performing daily tasks, extracting relevant clips based on natural language commands. This process involves converting video frames into sparse 3D keypoints using DINO features, which capture semantic and geometric scene information. Hand trajectories are estimated with HaMeR, providing detailed action representations. The retrieved clips serve as exemplars for the second stage—immediate behavior imitation.
In the execution phase, the Keypoint Action Tokens (KAT) model, a few-shot in-context learner, takes these examples and, conditioned on real-time visual input, generates precise robot action trajectories. This approach circumvents the need for costly fine-tuning, enabling robots to perform new tasks instantly. Experimental results across 12 household tasks demonstrate that R+X achieves an average success rate of 85%, outperforming baseline models that require retraining or fine-tuning. The system shows remarkable generalization to unseen objects, complex instructions, and cluttered environments, highlighting its robustness and scalability.
The significance of this work lies in its ability to democratize robot learning by utilizing naturally occurring videos, which are abundant and easy to collect. This reduces the barrier to deploying robots in real-world settings, making autonomous learning more accessible and cost-effective. The framework’s modular design, combining retrieval, perception, and imitation, offers a flexible blueprint for future research. It aligns with the broader vision of embodied AI, where robots continuously learn from everyday human activities, adapting to new tasks and environments without explicit supervision.
Looking ahead, future work will focus on improving retrieval accuracy in more dynamic scenes, integrating additional sensory modalities, and enabling online continual learning. The ultimate goal is to develop robots that can autonomously acquire a wide repertoire of skills from the vast, unstructured data generated by human daily life, transforming industries and daily living alike. This research marks a significant step toward realizing truly autonomous, adaptable robots capable of learning from the world around them in a manner akin to humans.
Deep Dive
Abstract
We present R+X, a framework which enables robots to learn skills from long, unlabelled, first-person videos of humans performing everyday tasks. Given a language command from a human, R+X first retrieves short video clips containing relevant behaviour, and then executes the skill by conditioning an in-context imitation learning method (KAT) on this behaviour. By leveraging a Vision Language Model (VLM) for retrieval, R+X does not require any manual annotation of the videos, and by leveraging in-context learning for execution, robots can perform commanded skills immediately, without requiring a period of training on the retrieved videos. Experiments studying a range of everyday household tasks show that R+X succeeds at translating unlabelled human videos into robust robot skills, and that R+X outperforms several recent alternative methods. Videos and code are available at https://www.robot-learning.uk/r-plus-x.
References (20)
Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
Norman Di Palo, Edward Johns
Deep ViT Features as Dense Visual Descriptors
Shirzad Amir, Yossi Gandelsman, Shai Bagon et al.
SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning
Jianlan Luo, Zheyuan Hu, Charles Xu et al.
Human-to-Robot Imitation in the Wild
Shikhar Bahl, Abhi Gupta, Deepak Pathak
Image Segmentation Using Text and Image Prompts
Timo Lüddecke, Alexander S. Ecker
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0
A. Padalkar, A. Pooley, Ajinkya Jain et al.
Gaussian Splatting SLAM
Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly et al.
ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection
Arpit Bahety, Priyanka Mandikal, Ben Abbatematteo et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Machel Reid, N. Savinov, Denis Teplyashin et al.
Concept2Robot: Learning manipulation concepts from instructions and human demonstrations
Lin Shao, Toki Migimatsu, Qiang Zhang et al.
Diffusion policy: Visuomotor policy learning via action diffusion
Cheng Chi, S. Feng, Yilun Du et al.
Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos
Annie S. Chen, Suraj Nair, Chelsea Finn
DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models
Norman Di Palo, Edward Johns
Open Source
Mick Watson
Emerging Properties in Self-Supervised Vision Transformers
Mathilde Caron, Hugo Touvron, Ishan Misra et al.
DITTO: Demonstration Imitation by Trajectory Transformation
Nick Heppert, Max Argus, T. Welschehold et al.
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Chen Wang, Linxi (Jim) Fan, Jiankai Sun et al.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Nils Reimers, Iryna Gurevych
A Comprehensive Survey of Continual Learning: Theory, Method and Application
Liyuan Wang, Xingxing Zhang, Hang Su et al.