Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
Retrieval-augmented policy enables zero-shot extension of vision-language-action models at test time, reducing costs by replacing fine-tuning with retrieval.
Key Findings
Methodology
This paper introduces RECAP, a retrieval-conditioned residual policy built upon the Cosmos Policy framework. During training, paired demonstrations from a source (pool) and target (query) embodiment are used to learn a residual action correction, with the core model frozen thereafter. At inference, the policy conditions on retrieved trajectories from a growing demonstration pool, enabling task extension without parameter updates. The approach integrates a world action model (WAM) with a future image prediction objective, reinforcing visual consistency and long-horizon behavior. Key components include: • training on paired demonstrations to learn residual corrections; • dynamic retrieval of matching trajectories during deployment; • combining retrieval with future image prediction for robust action generation.
Key Results
- In the PushT benchmark, expanding the retrieval pool at test time increased success rate from 6.0% to 34.9%, demonstrating 18× cost savings;
- On RoboTwin 2.0, success rates improved from 9.0% to 31.5% with pool growth, outperforming baseline cross-embodiment methods;
- Real robot experiments showed that using human demonstrations as retrieval pool enabled zero-shot transfer to unseen tasks, achieving 80% success in placing bottles and 30% in closing cabinets, validating practical applicability.
Significance
This work addresses the high costs of task-specific data collection and fine-tuning in robotic policies. By leveraging retrieval over a reusable demonstration pool, it enables rapid, cost-effective generalization to new tasks and embodiments. The approach significantly reduces the need for expensive teleoperation and training, making robot deployment more scalable and adaptable. The integration of a world action model with retrieval not only improves long-horizon planning but also enhances visual consistency, crucial for real-world applications. Overall, this paradigm shift from parameter tuning to data indexing opens new avenues for scalable autonomous systems in industry and service domains.
Technical Contribution
The key technical innovation is the combination of a frozen, retrieval-conditioned residual policy with a pre-trained world action model that incorporates future image prediction. This design allows task and embodiment transfer via indexing, avoiding costly fine-tuning. The residual formulation narrows the learning problem to embodiment-specific corrections, while the retrieval mechanism supplies high-level motion cues. The model’s training jointly optimizes action and scene prediction, ensuring grounded behavior. During inference, the dynamic retrieval process enables continuous task expansion, supported by a similarity metric over state and visual features. This framework extends the capabilities of existing vision-language-action models, enabling zero-shot generalization across unseen tasks and embodiments.
Novelty
This research is the first to embed a retrieval mechanism within a vision-language-action policy, replacing traditional fine-tuning for new tasks and entities. Unlike prior methods that rely solely on task-specific demonstrations or language prompts, RECAP leverages a reusable demonstration pool, enabling rapid adaptation without retraining. Its residual action formulation and integration with a world action model that predicts future images constitute a novel architecture, especially effective for long-horizon and cross-embodiment tasks. The approach bridges the gap between large-scale pretraining and practical deployment, setting a new standard for scalable robot generalization.
Limitations
- The method assumes shared low-level action spaces between query and pool embodiments; significant structural differences limit applicability.
- Retrieval quality heavily depends on the representational capacity of the demonstration descriptors; video-only sources require additional lifting into state-action space.
- In highly dynamic or large-scale embodiment gaps, residual predictions may accumulate errors, reducing accuracy. Further robustness mechanisms are needed.
Future Work
Future directions include developing embodiment-agnostic interfaces or learned translation modules to handle diverse action spaces, integrating more scalable and invariant representations for retrieval, and expanding retrieval sources to in-the-wild videos. Enhancing robustness against temporal and dynamical mismatches, as well as scaling to large, uncurated datasets, will be crucial for broad real-world deployment. Additionally, combining this approach with online learning and continual adaptation could further improve autonomous robot capabilities.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭，平时用的锅和刀都很熟悉，但有时候你会用不同的锅或刀子。一个聪明的助手可以帮你找到最合适的工具，只要你告诉它你想做什么。这个助手不用每次都重新学习，只要记住了各种工具的使用方法，就能根据你的需求快速帮你找到合适的工具。这样，你就不用每次都花时间重新训练它，它只需要从已有的工具库里检索信息，然后帮你完成任务。这个方法就像机器人一样，利用已有的示范和记忆，快速适应新任务，节省了很多时间和精力。
ELI14 Explained like you're 14
想象你在学校的美术课上画画，你平时用的画笔和颜料都很熟悉。有时候老师会让你用不同的画笔或者颜料，但你不用重新学怎么用，只要你告诉老师你想画什么，老师就会帮你找到之前用过的画笔和颜料，然后帮你继续画。这个过程就像机器人用的检索系统，它不用每次都重新学习新任务，而是从记忆里找出最合适的示范，然后根据这个示范调整自己的动作。这样，机器人可以很快学会新任务，就像你用熟悉的工具画画一样简单。这个方法让机器人变得更聪明、更快，能在不同的场景中帮你完成各种任务，就像你的老师帮你找到最合适的画笔一样。
Abstract
Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.