BiGraspFormer: End-to-End Bimanual Grasp Transformer
BiGraspFormer is an end-to-end transformer framework that directly generates coordinated bimanual grasps from point clouds, achieving high success and efficiency.
Key Findings
Methodology
BiGraspFormer employs a Single-Guided Bimanual (SGB) strategy, where a transformer decoder first generates diverse single-arm grasp candidates. These features are then integrated via specialized attention mechanisms to jointly predict bimanual poses and quality scores. The architecture includes an object encoder (PointNet+++Transformer), a single grasp proposer (DETR-style), a bimanual pair matcher (using quality metrics and collision checks), and a bimanual grasp generator with SGB attention. During training, bipartite matching aligns predictions with ground truth, optimizing position and quality losses to ensure coordination and stability.
Key Results
- On the DA2 dataset, success rate reaches 59.72% under disturbance conditions, outperforming previous methods like CGDF+distance (36.46%). In simulation, success rates exceed 89% in normal conditions, with inference times under 0.05 seconds, suitable for real-time deployment.
- The approach demonstrates superior diversity and robustness, with a 30% increase over baselines, effectively handling external disturbances and complex geometries. Real robot tests show over 80% success across various objects, confirming practical viability.
- Ablation studies reveal that the full SGB framework with attention modules yields the best performance, validating the importance of feature-guided coordination in high-DoF bimanual grasping.
Significance
This work advances robotic manipulation by enabling end-to-end, real-time bimanual grasping directly from raw point clouds. It addresses key challenges like collision avoidance, force balance, and coordination, which are critical for handling large, complex objects in industrial and service settings. The method's efficiency and robustness open new avenues for autonomous assembly, logistics, and assistive robotics, reducing reliance on handcrafted heuristics and modular pipelines.
Technical Contribution
The paper introduces a novel transformer-based architecture that unifies grasp generation and evaluation, leveraging single-arm features to guide dual-arm coordination. The SGB attention mechanism effectively reduces the search space from 12-DoF to manageable subproblems, enabling fast inference. The integration of multi-scale local and global features enhances the model's understanding of complex geometries, setting a new standard for end-to-end bimanual grasping.
Novelty
This is the first work to propose an end-to-end transformer framework for coordinated bimanual grasping directly from point clouds. Its key innovation lies in the SGB strategy, which uses single-arm grasp features to guide the joint prediction of dual-arm poses, significantly improving diversity, stability, and computational efficiency over prior modular or generative-evaluation methods.
Limitations
- The model's performance degrades in scenarios with severe occlusion or extremely cluttered environments due to insufficient point cloud information.
- Dependence on high-quality point cloud data and computational resources limits deployment on low-power platforms.
- Current training relies heavily on annotated datasets; unsupervised or semi-supervised approaches are needed for broader generalization.
Future Work
Future research will focus on integrating tactile and visual feedback for dynamic adjustment, exploring reinforcement learning for adaptive grasping, and reducing computational costs for embedded systems. Extending the framework to multi-object scenarios and real-time scene understanding will further enhance its industrial applicability.
AI Executive Summary
Robotic manipulation of large and complex objects remains a significant challenge, especially when coordination between two arms is required. Traditional methods often rely on modular pipelines, where grasp generation and evaluation are separate, leading to inefficiencies and coordination issues. BiGraspFormer addresses this gap by proposing a unified, end-to-end transformer-based framework capable of directly predicting coordinated bimanual grasps from raw point clouds.
The core innovation lies in the Single-Guided Bimanual (SGB) strategy. Initially, a transformer decoder generates diverse single-arm grasp candidates, leveraging learned features. These features are then fused through specialized attention mechanisms, enabling the model to jointly predict dual-arm poses and grasp quality scores. This approach effectively reduces the 12-DoF search space, simplifies the complex coordination task, and ensures stable, collision-free grasps.
Extensive experiments in simulation and real-world environments demonstrate the superiority of BiGraspFormer. Success rates under disturbance conditions reach 59.72%, with inference times below 0.05 seconds, outperforming state-of-the-art methods like CGDF and DPN-GPD. The model also exhibits high diversity, covering larger object surface areas, which is crucial for practical applications.
In real robot tests with UR5e arms, the system achieved over 80% success across various objects, validating its robustness and applicability. The framework's efficiency and stability suggest promising deployment in industrial automation, logistics, and assistive robotics. Future work aims to incorporate multi-modal sensing, reinforcement learning, and scene understanding to further enhance performance in dynamic, cluttered environments.
Deep Analysis
Background
机器人抓取技术经历了从传统几何模型到深度学习的演变。早期方法依赖CAD模型和手工规则，难以应对未知复杂对象。近年来，PointNet、GPD等模型推动了单臂6自由度抓取的突破，但在多臂场景中存在协调难题。双臂抓取涉及更高自由度（12自由度）和碰撞、力平衡等复杂约束，现有方案多采用模块化设计，效率低、协调性差。DA2数据集的出现推动了双臂抓取研究，但多为离线评估，缺乏端到端解决方案。整体来看，单臂技术已较成熟，但双臂协调仍亟待创新，尤其在实时性和多样性方面。
Core Problem
双臂抓取的核心难题在于高维动作空间（12自由度）带来的搜索复杂性，以及碰撞避免、力平衡和协调控制的多重约束。现有方法多采用分步处理或依赖外部生成器，导致效率低、协调性差，难以满足工业自动化对高效、稳定操作的需求。如何在保证多样性和稳定性的同时，实现端到端的快速预测，是当前亟需解决的难题。
Innovation
本研究提出端到端变换器架构，结合单-引导策略，创新性地利用单臂抓取特征指导双臂生成，显著降低搜索空间复杂度。引入多尺度特征融合和注意力机制，增强双臂协调能力，避免依赖外部生成器。模型实现全流程预测，提升效率和鲁棒性，首次实现从点云直接生成高质量双臂抓取方案，突破传统模块化方案的瓶颈。
Methodology
- �� 物体编码器：结合PointNet++提取局部几何特征，使用Transformer编码全局关系。• 单抓提议器：基于DETR架构，利用学习的查询生成多样单臂抓取候选。• 双臂匹配器：结合质量指标和碰撞检测，筛选合理的双臂对。• 双臂生成器：引入SGB注意力机制，结合单臂特征，联合预测双臂姿态和质量。• 损失函数：结合位置回归和质量评分，采用匈牙利匹配优化预测与真实对齐。• 训练流程：端到端优化，利用大量标注数据提升泛化能力。
Experiments
在DA2数据集上进行训练，采用模拟环境中的成功率和多样性指标进行评估。对比多种SOTA方法，验证在扰动和正常条件下的性能。采用Isaac Gym模拟，测试不同复杂度对象的抓取效果。实机器人实验中，利用UR5e机器人在多姿态、多对象场景下验证模型的实用性和鲁棒性。通过ablation研究验证模型关键模块的贡献。
Results
在扰动条件下，成功率达59.72%，优于CGDF+距离策略36.46%；推理时间小于0.05秒，适合实时应用。模拟成功率在正常条件下达89.67%，远超对比方法。多样性指标提升30%以上，表现出良好的覆盖性和稳定性。实机器人实验中，成功率超过80%，在复杂环境中表现出强鲁棒性，验证了模型的实用价值。
Applications
该技术适用于仓储、制造、救援等场景，实现自主搬运和装配。无需复杂的预定义模型，适应多样未知对象，提升工业自动化水平。未来还可结合视觉和触觉信息，拓展到动态环境中的自主操作。
Limitations & Outlook
模型在遮挡严重或极端复杂场景下仍有失败，主要因点云信息不足。对硬件依赖较强，需优化传感和计算平台。训练依赖大量标注数据，未来需探索弱监督或无监督学习以提升泛化能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，准备多种食材。每次你都先用手感受食材的形状和位置，然后决定用哪只手去拿。你会先试试用左手抓一块肉，再用右手抓一块蔬菜，确保两只手不会碰到对方，也不会掉下来。这个过程就像机器人在处理大块物体时，要同时用两只“手”抓稳，避免碰撞或用力不均。BiGraspFormer就像一个聪明的厨师，它能快速判断每只手应该怎么拿，确保两只手配合得天衣无缝，既稳又高效。它不用事先准备好所有方案，而是边看边决定，像你在厨房里灵活应对各种食材一样，机器人也能应对各种复杂的物体。
ELI14 Explained like you're 14
想象你在玩拼图游戏，你要用两只手同时拼两个不同的拼块。刚开始，你会先用一只手试试哪个拼块能稳稳地拼上去，然后用另一只手找到合适的位置。你会不断调整，确保两个拼块都稳固，不会掉下来，也不会碰到对方。这就像机器人在抓大东西时，要用两只“手”合作，既要稳，还不能撞到一起。BiGraspFormer就像一个超级聪明的拼图高手，它能一眼看出怎么用两只手把东西稳稳地拿起来，还能快得像闪电一样。它不用事先告诉你怎么拼，而是根据物体的形状，自己快速决定怎么抓，确保每次都成功。这样，机器人就可以像人一样灵活、快速地完成复杂的任务了！
Abstract
Bimanual grasping is essential for robots to handle large and complex objects. However, existing methods either focus solely on single-arm grasping or employ separate grasp generation and bimanual evaluation stages, leading to coordination problems including collision risks and unbalanced force distribution. To address these limitations, we propose BiGraspFormer, a unified end-to-end transformer framework that directly generates coordinated bimanual grasps from object point clouds. Our key idea is the Single-Guided Bimanual (SGB) strategy, which first generates diverse single grasp candidates using a transformer decoder, then leverages their learned features through specialized attention mechanisms to jointly predict bimanual poses and quality scores. This conditioning strategy reduces the complexity of the 12-DoF search space while ensuring coordinated bimanual manipulation. Comprehensive simulation experiments and real-world validation demonstrate that BiGraspFormer consistently outperforms existing methods while maintaining efficient inference speed (<0.05s), confirming the effectiveness of our framework. Code and supplementary materials are available at https://sites.google.com/view/bigraspformer