LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer
LAP enables zero-shot cross-embodiment robot control by representing low-level actions as natural language, achieving over 50% success rate.
Key Findings
Methodology
LAP reformulates robot low-level actions as natural language descriptions (e.g., 'move forward 5cm'), aligning supervision with pre-trained vision-language model distributions. It employs structured templates to generate language actions without extra annotation or architecture changes, integrating a hybrid Transformer architecture that combines a vision-language backbone with a lightweight flow-matching action expert for real-time control. During training, the model optimizes on large-scale datasets like Open X-Embodiment and MolmoAct, jointly learning action prediction and visual question answering (VQA). This approach maintains semantic richness, reduces distribution mismatch, and enhances cross-embodiment generalization, enabling zero-shot transfer to unseen robots.
Key Results
- LAP-3B achieves over 50% success rate on unseen robot embodiments (YAM, Kinova, Franka) across tasks like pick-and-place, sorting, pouring, outperforming prior VLAs by approximately 2×. It demonstrates strong zero-shot transfer, with fewer data needed for fine-tuning, and scales well with model size. Ablation studies confirm that natural language action descriptions provide better semantic grounding than continuous or tokenized actions, facilitating cross-embodiment generalization.
- Across multiple experiments, the model maintains high performance without embodiment-specific fine-tuning, showing robustness in real-world manipulation tasks. Combining VQA with action prediction further improves accuracy, indicating the benefits of multi-task learning. The results validate that structured language actions serve as an effective universal interface for robot control.
- The experiments highlight that natural language-based supervision mitigates distributional shifts, improves transferability, and supports efficient adaptation, marking a significant step toward generalist robotic policies.
Significance
This work addresses a fundamental challenge in robotics: enabling models to operate seamlessly across diverse robot morphologies without extensive retraining. By representing actions as natural language, LAP leverages the semantic richness of pre-trained vision-language models, significantly improving zero-shot transfer. This innovation reduces deployment costs, accelerates robot adaptation, and broadens applicability in industrial, service, and domestic settings. It paves the way for more flexible, scalable, and intelligent robotic systems capable of understanding and executing complex tasks in varied environments, thus advancing both theoretical understanding and practical deployment of autonomous robots.
Technical Contribution
The core technical innovation is the introduction of language-based action supervision (LAP), which aligns robot actions with natural language, preserving semantic structure and enabling zero-shot transfer. The model integrates a vision-language backbone with a lightweight flow-matching control module, allowing efficient real-time execution. Unlike prior approaches relying on token-based or continuous action spaces, LAP's structured language representation maintains semantic consistency across different embodiments. The joint training on large-scale datasets with multi-task objectives (action prediction and VQA) enhances the model’s generalization and scalability, establishing a new paradigm for embodied AI.
Novelty
This is the first work to formalize low-level robot actions as natural language descriptions for pre-training, significantly improving cross-embodiment generalization. Unlike previous methods that depend on embodiment-specific conditioning or action space unification, LAP achieves zero-shot transfer solely through semantic language supervision. Its integration with large-scale datasets and hybrid architecture marks a novel approach, setting a new benchmark in vision-language embodied learning.
Limitations
- The structured natural language templates may limit expressiveness for highly complex or unstructured tasks, requiring future expansion of language generation capabilities.
- Generalization to highly dynamic or unstructured environments remains untested; robustness under extreme conditions needs further validation.
- Training large models with multi-task objectives incurs high computational costs, hindering real-time deployment in resource-constrained settings.
Future Work
Future research will explore more flexible language generation methods, including learned natural language descriptions, to handle complex actions. Extending the framework to multi-robot collaboration and multi-modal perception will be prioritized. Additionally, efforts to optimize model efficiency and reduce computational demands will be pursued, aiming for broader real-world applicability and deployment in embedded systems.
AI Executive Summary
Robotics has long aimed for a universal control policy capable of operating across diverse robot morphologies without extensive retraining. Traditional vision-language models (VLMs) have shown promise in understanding complex scenes and instructions but struggle to generalize to new robot forms. This limitation hampers the deployment of versatile robotic systems in real-world environments.
In this context, the authors introduce a novel approach called Language-Action Pre-training (LAP). The key idea is to represent robot low-level actions using natural language descriptions, such as 'move forward 5cm' or 'tilt right 45 degrees.' This structured language acts as a universal interface, aligning the supervision signal with the pre-trained VLM’s distribution. By doing so, the model maintains semantic richness and reduces distributional mismatch, which are critical for zero-shot generalization.
The implementation, LAP-3B, combines a vision-language backbone with a lightweight flow-matching control module, enabling real-time control at 25Hz. The training leverages large-scale datasets like Open X-Embodiment and MolmoAct, incorporating multi-task objectives such as action prediction and visual question answering (VQA). The results are impressive: in tests on unseen robot types and complex manipulation tasks, LAP-3B achieves success rates over 50%, roughly doubling the performance of previous state-of-the-art models. These findings demonstrate that natural language action representations significantly enhance the model’s ability to generalize across different robot embodiments.
This breakthrough has profound implications for robotics. It suggests that by embedding actions into a semantic language space, robots can adapt to new hardware configurations and tasks with minimal or no additional training. This reduces costs, accelerates deployment, and opens new avenues for autonomous multi-robot systems capable of flexible, intelligent behavior. Although promising, challenges remain, such as expanding language expressiveness and reducing computational costs. Future work will focus on these areas, aiming to realize truly generalist robots capable of understanding and executing a broad spectrum of tasks across environments.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
A long-standing goal in robotics is a generalist policy that can be deployed zero-shot on new robot embodiments without per-embodiment adaptation. Despite large-scale multi-embodiment pre-training, existing Vision-Language-Action models (VLAs) remain tightly coupled to their training embodiments and typically require costly fine-tuning. We introduce Language-Action Pre-training (LAP), a simple recipe that represents low-level robot actions directly in natural language, aligning action supervision with the pre-trained vision-language model's input-output distribution. LAP requires no learned tokenizer, no costly annotation, and no embodiment-specific architectural design. Based on LAP, we present LAP-3B, which to the best of our knowledge is the first VLA to achieve substantial zero-shot transfer to previously unseen robot embodiments without any embodiment-specific fine-tuning. Across multiple novel robots and manipulation tasks, LAP-3B attains over 50% average zero-shot success, delivering roughly a 2x improvement over the strongest prior VLAs. We further show that LAP enables efficient adaptation and favorable scaling, while unifying action prediction and VQA in a shared language-action format that yields additional gains through co-training.