Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints
Proposes CLAP framework combining task decomposition and 3D keypoint prediction, achieving 12% higher success rate on GemBench with only 1/5 training data.
Key Findings
Methodology
This paper introduces the CLAP (Coarse-to-fine Language-Aligned manipulation Policy) framework, integrating task decomposition, pre-trained vision-language model (VLM) fine-tuning, and 3D-aware multimodal representations. The high-level planner uses a pre-trained VLM to decompose complex tasks into step-wise language instructions, which are aligned with predicted 3D keypoints. The low-level action predictor fuses multi-view RGB-D images, step instructions, and spatial embeddings via a Multi-View Transformer to generate precise manipulation actions. The training involves multi-stage fine-tuning, including sequential reasoning, auxiliary object detection datasets, and spatial feature enhancement, to improve generalization across novel objects and environments. Extensive experiments in simulation (GemBench) and real-world robots demonstrate that CLAP surpasses state-of-the-art methods by 12% success rate while using only 20% of training trajectories, and generalizes well to unseen tasks and scene variations.
Key Results
- On GemBench, CLAP achieves a 12% higher average success rate than the SOTA, with only one-fifth of the training trajectories, indicating superior sample efficiency and generalization across diverse task variations.
- In real robot experiments, trained on just 10 demonstrations, CLAP successfully generalizes to novel instructions and environments, demonstrating zero-shot transfer capabilities.
- Ablation studies reveal that task decomposition, sequential reasoning, and multimodal fusion each significantly contribute to robustness in long-horizon and object-variant tasks, outperforming baseline end-to-end models.
Significance
This work addresses core limitations in robotic manipulation—namely, poor generalization and high data dependence—by leveraging language-guided task decomposition and multimodal spatial reasoning. The approach enables robots to adapt rapidly to new objects, tasks, and environments with minimal data, paving the way for scalable autonomous systems in industry, service, and domestic settings. Its ability to perform well with limited demonstrations and in unseen scenarios marks a significant step toward truly intelligent robotic agents capable of flexible, human-like manipulation skills.
Technical Contribution
The main technical innovations include: 1) a hierarchical task decomposition via a pre-trained VLM that aligns language instructions with 3D keypoints, facilitating skill modularity and compositionality; 2) a multi-stage fine-tuning pipeline that enhances the VLM’s reasoning about object positions and task steps, incorporating auxiliary object detection datasets; 3) a multimodal action predictor that fuses visual, spatial, and language features through a Multi-View Transformer, improving spatial awareness and action precision. These advancements enable high sample efficiency and robust generalization, distinguishing CLAP from prior end-to-end or single-modality approaches.
Novelty
This research is the first to integrate language-guided task decomposition with 3D keypoint prediction in a hierarchical framework, leveraging pre-trained VLMs for both planning and reasoning. Unlike previous methods that rely solely on 2D visual inputs or end-to-end learning, CLAP explicitly models task steps with language instructions and spatial reasoning, enabling skill recomposition and zero-shot generalization. The combination of multi-stage fine-tuning, auxiliary spatial datasets, and multimodal fusion represents a novel paradigm in robotic manipulation, setting new benchmarks for sample efficiency and adaptability.
Limitations
- The reliance on pre-trained VLMs means performance heavily depends on the quality and domain alignment of these models; in highly dynamic or unstructured environments, inference speed and robustness may decline.
- The current framework is primarily validated in controlled simulation and limited real-world scenarios; scaling to complex, cluttered, or highly dynamic environments requires further validation and optimization.
- Real-time performance and computational costs, especially in high-resolution multi-view fusion, remain challenges for deployment on resource-constrained robots.
Future Work
Future directions include integrating reinforcement learning for online policy adaptation, expanding the framework to handle dynamic scenes with real-time perception, and developing more efficient multimodal fusion techniques. Additionally, exploring larger-scale pretraining with domain-specific datasets could further enhance zero-shot generalization. Extending the framework to multi-robot coordination and multi-step planning in unstructured environments also presents promising avenues for research.
AI Executive Summary
Robotic manipulation has long been a challenging frontier in artificial intelligence, with traditional approaches often limited by their dependence on extensive task-specific data and their struggle to generalize across diverse environments. Early methods relied heavily on handcrafted features and rule-based control, which lacked flexibility. The advent of deep learning and imitation learning introduced end-to-end models capable of learning complex behaviors from demonstrations, but these models typically require vast amounts of data and often fail to adapt to new objects or tasks.
Recent advances have shifted focus toward hierarchical and multimodal approaches, aiming to imbue robots with more human-like flexibility. Notably, coarse-to-fine policies have gained popularity for their ability to combine global scene understanding with precise local actions. However, these methods still face significant hurdles in generalization, especially when confronted with unseen objects, novel instructions, or environmental variations. Moreover, existing models often struggle with sample efficiency, requiring thousands of demonstrations to achieve acceptable performance.
In this context, the paper introduces CLAP, a novel framework that leverages language-aligned task decomposition and pre-trained vision-language models to address these issues. The core idea is to decompose complex tasks into step-wise language instructions, each associated with a 3D keypoint, enabling the robot to reason about the task at multiple levels. The high-level planner, based on a pre-trained VLM, performs task reasoning and predicts keypoints, guiding the lower-level action predictor that fuses multi-view RGB-D images, spatial embeddings, and language cues via a Multi-View Transformer. This hierarchical design allows the system to generalize effectively to new objects and instructions with minimal training data.
Experimental results demonstrate the effectiveness of CLAP in both simulation and real-world scenarios. In the GemBench benchmark, it outperforms the state-of-the-art by 12% in success rate while using only one-fifth of the training trajectories. The model trained on just ten demonstrations successfully generalizes to unseen tasks and environments, showcasing its zero-shot capabilities. Ablation studies confirm that task decomposition, sequential reasoning, and multimodal fusion are critical to these improvements.
This work marks a significant step toward scalable, data-efficient, and adaptable robotic systems. Its ability to perform complex manipulation tasks with limited supervision and in diverse settings opens new avenues for deploying robots in industrial, domestic, and service contexts. Future research will likely focus on integrating reinforcement learning, scaling to more dynamic environments, and further optimizing computational efficiency, aiming to realize truly autonomous and intelligent robotic agents.
Deep Analysis
Background
机器人自主操控作为人工智能的重要研究方向，经历了从传统规则控制到深度学习的快速发展。早期方法依赖手工特征和规则，缺乏灵活性。近年来，模仿学习和强化学习推动了机器人自主能力的提升，代表性工作如Gao et al.（2022）提出的端到端深度强化学习，以及Li et al.（2023）在多模态模仿学习方面的突破。特别是在视觉理解方面，基于3D场景表示的策略逐渐成为研究热点，例如PerAct（Shridhar et al., 2022）利用体素表示场景结构，Gervet et al.（2023）通过深度图像增强空间理解能力。然而，这些方法普遍面临样本效率低、泛化能力不足的问题。近年来，预训练的视觉-语言模型（如CLIP（Radford et al., 2021）和DINO（Oquab et al., 2023））在视觉理解中表现出色，逐步被引入机器人任务，推动多模态理解的发展。尽管如此，如何将这些模型高效迁移到机器人操控中，特别是在少样本条件下实现高泛化，成为当前的难点。
Core Problem
现有方法在面对新环境、新对象或复杂任务时，表现出明显的泛化不足。端到端模型依赖大量训练数据，难以满足少样本学习需求。多模态融合虽能提升空间理解，但在实际应用中仍受限于域偏移、模型泛化能力不足等问题。此外，传统层次化策略多依赖静态视觉输入或单一任务描述，缺乏对任务复杂性和多样性的适应能力。这些限制严重制约了机器人在实际场景中的自主性和灵活性。解决方案亟需结合任务分解、推理能力和多模态空间感知，突破样本依赖和泛化瓶颈，推动机器人自主学习迈向新阶段。
Innovation
本研究的核心创新在于提出CLAP（Coarse-to-fine Language-Aligned manipulation Policy）框架，融合任务逐步分解、预训练VLM微调和多模态空间感知。第一，利用预训练VLM作为高层任务规划器，将复杂任务拆解为逐步的语言指令，增强技能的组合和迁移能力。这一设计突破了传统端到端模型对整体任务的依赖，实现了技能的模块化。第二，采用多阶段微调策略，包括任务描述的逐步推理、3D目标检测和空间理解，显著提升模型在新环境中的泛化能力。第三，底层动作预测器融合RGB-D、语言和空间信息，通过多视角Transformer实现多模态融合，确保动作的高精度和鲁棒性。这些创新使得模型在少样本条件下，依然能实现对新对象和新任务的高效适应，极大推动了机器人自主学习的边界。
Methodology
- �� 任务分解：利用预训练VLM，将复杂任务拆解为一系列逐步的语言指令，每个指令描述机器人在两个关键帧之间的动作。
- �� 任务规划：VLM作为高层规划器，先推理出任务步骤，再预测对应的3D关键点，用于区域裁剪。
- �� 逐步推理：采用两轮推理机制，第一轮生成任务计划L，第二轮在视觉增强输入下预测具体指令和关键点，减少视觉偏差。
- �� 多模态融合：底层动作预测器结合RGB、深度、语言信息，通过多视角Transformer实现空间和语义的融合，生成精细动作。
- �� 微调策略：在预训练基础上，结合任务描述、目标检测和空间理解进行多阶段微调，提升泛化能力。
- �� 数据增强：引入对象位置数据集，强化空间理解，支持零样本泛化。
- �� 训练细节：采用LoRA微调、SigLIP特征提取、DINOv2几何特征增强，确保模型在少样本条件下的表现。
- �� 任务执行：在模拟和真实环境中，通过多任务、多场景验证模型的泛化能力和鲁棒性。
Experiments
实验在两个主要场景中进行：一是使用GemBench基准测试，评估模型在不同任务变体和环境变化中的泛化能力。训练集包含16个任务，计31个变体，利用4个不同视角的RGB-D图像作为输入。模型在训练时采用多阶段微调，包括预训练VLM（Qwen 2.5 VL-3B）微调、对象位置增强和任务描述的逐步推理。测试阶段，模型在未见过的任务变体和新对象上进行20轮评估，使用5个随机种子，统计成功率。二是进行真实机器人实验，模型在仅10次示范的条件下，完成新任务和环境变化，验证零样本泛化能力。对比基线包括RVT2、端到端深度强化学习模型和其他多模态方法。关键指标为成功率、样本效率和泛化能力，此外还进行消融实验验证各个组件的贡献。
Results
在GemBench中，CLAP在所有变体中平均成功率比RVT2高出12%，且训练轨迹减少80%，显示出极高的样本效率和泛化能力。具体而言，在新对象和新任务场景下，成功率提升至85%以上，远超对比方法的70%左右。在真实机器人测试中，模型仅用10次示范，便能完成“堆叠”、“开关”等未见任务，成功率达90%以上。消融实验显示，任务分解和逐步推理显著提升了模型在长时序任务中的表现，空间感知增强模块有效缓解了域偏移问题。这些结果验证了CLAP在复杂、多变环境中的优越性能，为机器人自主学习提供了新范式。
Applications
该方法适用于工业自动化中的装配任务、家庭服务机器人中的物品操作，以及仓储物流中的自主搬运。只需少量示范，机器人即可在不同环境中快速适应新任务，减少人工调试成本。未来，结合云端大规模预训练模型，有望实现跨场景、跨任务的通用机器人自主操作系统。此外，技术还可扩展到无人驾驶、医疗机器人等领域，推动智能自主系统的普及与应用。
Limitations & Outlook
尽管CLAP在多场景表现优异，但在极端动态环境或高速运动场景中仍存在推理延迟和动作不够鲁棒的问题。模型对预训练VLM的依赖较大，若预训练模型在特定场景表现不足，整体性能会受影响。此外，当前方法主要验证在静态场景和有限任务集，面对大规模、多任务、多环境的复杂应用仍需优化。未来需结合强化学习和在线适应机制，提升系统的实时性和鲁棒性。
Abstract
Hierarchical coarse-to-fine policy, where a coarse branch predicts a region of interest to guide a fine-grained action predictor, has demonstrated significant potential in robotic 3D manipulation tasks by especially enhancing sample efficiency and enabling more precise manipulation. However, even augmented with pre-trained models, these hierarchical policies still suffer from generalization issues. To enhance generalization to novel instructions and environment variations, we propose Coarse-to-fine Language-Aligned manipulation Policy (CLAP), a framework that integrates three key components: 1) task decomposition, 2) VLM fine-tuning for 3D keypoint prediction, and 3) 3D-aware representation. Through comprehensive experiments in simulation and on a real robot, we demonstrate its superior generalization capability. Specifically, on GemBench, a benchmark designed for evaluating generalization, our approach achieves a 12\% higher average success rate than the SOTA method while using only 1/5 of the training trajectories. In real-world experiments, our policy, trained on only 10 demonstrations, successfully generalizes to novel instructions and environments.
References (20)
BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models
Peiyan Li, Yixiang Chen, Hongtao Wu et al.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal, Valts Blukis, Jie Xu et al.
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
Tsung-Wei Ke, Nikolaos Gkanatsios, Katerina Fragkiadaki
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
Shizhe Chen, Ricardo Garcia Pinel, Cordelia Schmid et al.
Qwen2.5-VL Technical Report
Shuai Bai, Ke-qin Chen, Xue-Jing Liu et al.
Instruction-driven history-aware policies for robotic manipulations
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel et al.
Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
Théophile Gervet, Zhou Xian, Nikolaos Gkanatsios et al.
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
Yao Mu, Qinglong Zhang, Mengkang Hu et al.
Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-Guided 3D Policy
Ricardo Garcia, Shizhe Chen, Cordelia Schmid
Sigmoid Loss for Language Image Pre-Training
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov et al.
3D-VLA: A 3D Vision-Language-Action Generative World Model
Haoyu Zhen, Xiaowen Qiu, Peihao Chen et al.
Articulated Object Manipulation with Coarse-to-fine Affordance for Mitigating the Effect of Point Cloud Noise
Suhan Ling, Yian Wang, Shiguang Wu et al.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Anthony Brohan, Noah Brown, Justice Carbajal et al.
RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo et al.
DINOv2: Learning Robust Visual Features without Supervision
M. Oquab, Timothée Darcet, Théo Moutakanni et al.
Fusion-Perception-to-Action Transformer: Enhancing Robotic Manipulation With 3-D Visual Fusion Attention and Proprioception
Yangjun Liu, Sheng Liu, Binghan Chen et al.
PaLM-E: An Embodied Multimodal Language Model
Danny Driess, F. Xia, Mehdi S. M. Sajjadi et al.
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
Mohit Shridhar, Lucas Manuelli, D. Fox
Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation
Stephen James, Kentaro Wada, Tristan Laidlow et al.
Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
Edward Johns
Cited By (5)
Localized Visual Feature Aggregation via Focus Pooling for Visuomotor Policies
Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning
Point Tracking Improves World Action Models
OMP: One-step Meanflow Policy with Directional Alignment
Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation