Gripper-aware Vision Language Action Models
Proposes GVLA, integrating multi-gripper encoding and adapter routing, trained on MiGA dataset with 103,000 demonstrations, achieving 7.62% improvement over baselines.
Key Findings
Methodology
The GVLA framework combines a novel multi-gripper tokenizer with a dual Mixture-of-Adapters (MoA) mechanism within a Transformer backbone. The multi-gripper tokenizer encodes gripper information at platform, type, and instance levels, producing structured embeddings that facilitate strategy differentiation. The dual MoA routes action generation through gripper-specific and platform-specific experts, enabling efficient parameter sharing and specialization. The large-scale MiGA dataset, with 103,000 demonstrations across five gripper types, provides rich supervision for training. The model employs a conditional flow matching loss for action prediction and an auxiliary gripper classification loss, optimizing end-to-end for strategy-aware manipulation. Extensive experiments in simulation and real robots demonstrate superior performance and generalization.
Key Results
- GVLA outperforms baselines by 7.62% in success rate across multiple tasks. It exhibits strong zero-shot and few-shot adaptation, with success rate increases of 15% after minimal fine-tuning on new objects. In complex multi-object scenarios, success rates reach over 85%, significantly higher than models ignoring gripper differences. Ablation studies confirm that multi-gripper encoding and MoA contribute 4-5% each to performance gains.
- In transfer tasks, GVLA maintains high success rates on unseen objects and tasks, with minimal data, demonstrating robust generalization. The model's strategy routing effectively adapts to diverse gripper configurations, enabling seamless cross-gripper operation. The dataset's diversity and the structured embedding approach underpin these results.
- Ablation results reveal that removing multi-gripper tokens or MoA reduces success by 4-6%, highlighting their importance. The model also shows improved efficiency in real-world experiments, with faster convergence and higher success rates compared to prior methods.
Significance
This work advances the field of robotic manipulation by explicitly modeling embodiment-dependent strategies, addressing a key limitation of prior vision-language models. The integration of structured gripper encoding and adaptive routing enables robots to handle a variety of grippers and tasks with high accuracy and robustness. The MiGA dataset provides a valuable resource for future research, fostering development of more generalizable, embodiment-aware policies. This approach paves the way for versatile, autonomous robots capable of operating in unstructured environments with multiple tools and configurations, significantly impacting industrial automation, service robotics, and human-robot collaboration.
Technical Contribution
The paper introduces a multi-gripper tokenizer that encodes gripper information at multiple levels, capturing morphology and constraints. The dual MoA mechanism allows efficient, parameter-sharing strategy modulation conditioned on gripper type and platform, enabling high flexibility with fewer parameters. The large-scale MiGA dataset offers diverse, annotated demonstrations across five gripper types, supporting strategy learning and transfer. These innovations collectively enable a new class of embodiment-aware, strategy-conditioned vision-language models, surpassing existing approaches that treat grippers as invariant or rely on geometric transfer alone.
Novelty
This is the first work to systematically incorporate multi-level gripper encoding and adaptive expert routing within a vision-language framework for robotic manipulation. Unlike prior studies focusing solely on grasp pose transfer or geometric adaptation, this approach explicitly models strategy divergence induced by different gripper embodiments. The combination of structured gripper tokens, dual MoA, and a comprehensive dataset represents a significant step forward in embodiment-aware manipulation learning, enabling robots to generalize across diverse tools and tasks.
Limitations
- Despite improvements, the model's performance drops in highly cluttered or dynamic environments where strategy complexity increases significantly. The dataset, while large, may not cover all possible gripper morphologies or novel tools, limiting generalization. Computational costs remain high for training and inference, especially in real-time applications. Future work should focus on model compression, multi-modal integration, and broader dataset expansion to address these issues.
Future Work
Future directions include integrating tactile and force feedback for finer control, developing more efficient architectures for real-time deployment, and expanding the dataset to include more diverse gripper types and complex tasks. Additionally, exploring unsupervised or semi-supervised learning for strategy discovery and transfer across robotic platforms will further enhance embodiment-aware manipulation capabilities.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭，不同的厨具（刀、勺子、锅）需要不同的操作方法。以前的机器人就像只会用一种厨具，做所有菜都一样，但这样很有限。现在，我们教它认识各种厨具，知道用不同的方法做不同菜。比如，用刀切菜，用勺子舀汤，用锅炒菜。我们还准备了很多示范视频，告诉它每种厨具怎么用，怎么操作。这样，机器人就能根据不同的厨具，灵活地完成各种任务。它变得更聪明、更灵活，能应对各种厨房挑战，就像一个真正的厨师一样。未来，它还能学习新厨具，快速适应新菜谱，变得更厉害！
ELI14 Explained like you're 14
想象你在学校的科学实验室里做实验，不同的工具（比如烧杯、显微镜、火焰）需要不同的操作方式。以前的机器人就像只会用一种工具，做所有实验都一样，但这样很受限制。现在，我们教它认识各种工具，知道用不同的方法完成不同的任务。比如，用烧杯倒水，用显微镜观察细胞，用火焰加热。我们还给它很多示范视频，告诉它每种工具怎么用，怎么操作。这样，机器人就能根据不同的工具，灵活地完成各种实验任务。它变得更聪明了，也更像一个真正的科学家助手。未来，它还能学会新工具，快速适应新任务，就像我们学新技能一样，变得越来越厉害！
Abstract
Vision language action models (VLAs) have advanced general purpose robotic grasping and manipulation by enabling robots to interpret visual observations and natural language instructions to generate executable action sequences. However, existing VLAs often implicitly assume gripper invariance, despite grasping strategies being inherently embodiment-dependent. Different gripper types, such as parallel-jaw and suction, usually require distinct interaction strategies to achieve the same grasping objective. Moreover, current datasets for VLAs predominantly rely on parallel-jaw grippers, limiting gripper-aware learning. To address this gap, we introduce MiGA, a multi-gripper-aware dataset spanning five distinct gripper types across multiple robots with 103,000 demonstrations, explicitly capturing strategy divergence under shared task objectives. We further propose GVLA, which combines a new multi-gripper tokenizer with adapter-based policy routing. Our new gripper encoding induces structured embedding information that balances parameter sharing and strategy differentiation, while layer-wise probing confirms meaningful gripper-conditioned representations for VLAs. Intensive experiments in both simulation and real-world robots show that our GVLA outperforms the current baselines across evaluated settings. Our method also improves zero-shot generalization or few-shot adaptation to new objects or unseen tasks, and enable more efficient gripper adaptation.