Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning
Hierarchical VLM-guided semantic-geometric framework improves task-aware bimanual grasping, achieving 88.9% strategy alignment in real-world tests.
Key Findings
Methodology
This paper introduces a hierarchical framework combining multi-view RGB-D fusion, global 6-DoF grasp candidate generation via AnyGrasp, and VLM-based semantic reasoning for task-conditioned affordance filtering. The process begins with multi-view sensors capturing overlapping RGB-D data, fused into a dense scene point cloud. A category-agnostic grasp generator proposes diverse grasp poses. VLM (e.g., GPT-5) analyzes task descriptions and object detections to determine arm strategies (single or dual), then identifies task-relevant contact regions through language prompts. These regions are projected into 3D space to filter grasp candidates, ensuring geometric feasibility aligned with task semantics. The approach operates without category-specific training, enabling flexible, task-aware bimanual manipulation in unstructured environments.
Key Results
- Across nine real-world tasks, strategy alignment reached 88.9%, outperforming geometric-only (9.0%) and semantic-only baselines (55.6%). The method excels in multi-object coordination, tool use, and human-robot handover, with over 30% improvement in success rates. Ablation studies confirmed the importance of multi-modal reasoning, showing that combining semantic and geometric filtering yields the best performance.
- In complex tasks like tool manipulation and object transfer, success rates increased significantly, demonstrating robustness. The approach adapts to diverse object sizes and task demands, maintaining high strategy consistency.
- The experiments validate that explicit semantic reasoning over affordances and arm allocation is critical for reliable, strategy-aligned bimanual manipulation, especially in cluttered, dynamic environments.
Significance
This work advances robotic manipulation by integrating large-scale vision-language models for task-conditioned affordance understanding, overcoming the limitations of purely geometric methods. It addresses longstanding challenges in multi-object, multi-arm coordination, enabling robots to interpret complex instructions and execute them reliably. The approach paves the way for more autonomous, intelligent robots capable of operating seamlessly in unstructured, real-world settings, bridging the gap between perception, reasoning, and action. Its generalization without category-specific training marks a significant step toward versatile robotic systems adaptable to diverse tasks and environments.
Technical Contribution
The paper introduces a novel hierarchical framework that synergizes multi-view scene understanding, zero-shot semantic reasoning via VLM, and geometric grasp filtering. Key innovations include the use of language prompts for arm strategy and contact region determination, spatial projection for region filtering, and an integrated pipeline that ensures semantic-task alignment with geometric feasibility. This approach extends the capabilities of existing grasping methods by enabling task-aware, strategy-consistent bimanual manipulation without requiring extensive training data, thus broadening the application scope of VLMs in robotics.
Novelty
This is the first work to leverage large-scale vision-language models for explicit, task-conditioned region and arm strategy reasoning in bimanual manipulation. Unlike prior methods focusing solely on object part segmentation or single-arm grasping, it introduces a hierarchical, multi-modal approach that jointly reasons about contact regions, arm allocation, and task semantics. Its zero-shot generalization capability and real-world validation distinguish it from previous category-dependent or dataset-specific solutions, marking a significant innovation in semantic robotic grasping.
Limitations
- The approach relies on high-quality multi-view perception, which can be affected by occlusions, sensor noise, and dynamic scene changes, potentially reducing robustness in cluttered or highly dynamic environments.
- VLM inference introduces computational overhead, limiting real-time performance. Optimization for faster reasoning is necessary for deployment in time-critical applications.
- Generalization to unseen tasks or novel object categories remains limited; future work should focus on improving adaptability and robustness through continual learning or domain adaptation techniques.
Future Work
Future directions include integrating reinforcement learning to optimize manipulation strategies, enhancing real-time inference through model compression, and extending the framework to multi-robot coordination. Additionally, exploring adaptive prompts for VLM to handle more complex, multi-step tasks and environments will further improve autonomy and robustness.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房帮忙做饭。你需要用两只手同时完成不同的任务，比如一只手拿刀切菜，另一只手拿锅搅拌。机器人也是这样，但它要先看清场景，理解每个工具的用途，然后决定用哪只手去操作哪个地方。传统的方法就像盯着刀和锅的形状，看看能不能用；而这项新技术还会考虑菜的用途，比如切菜是为了做汤，搅拌是为了让味道更均匀。它会用“眼睛”和“脑袋”一起工作，理解任务的意思，然后找到最合适的地方去操作。这样，机器人就能像厨师一样，聪明地完成复杂的任务，不仅仅是看着形状，还能理解“为什么”和“怎么做”。
ELI14 Explained like you're 14
想象你在厨房帮妈妈做饭。你知道每个工具的用处，比如刀用来切菜，锅用来煮汤。机器人也是这样，但它要用“眼睛”看场景，用“脑袋”理解任务。以前的机器人只会盯着物体的形状，像看着一把刀就知道要切菜，但不知道这是不是做汤的关键。现在，这个新方法让机器人可以理解任务的意思，比如“帮我把汤倒进碗”，然后它会找出合适的区域去操作，比如找到锅的把手或汤的出口。它还会用两只手合作，比如一只手稳住锅，另一只手倒汤。这样，机器人就变得更聪明、更像人类厨师，能在厨房里完成复杂的任务啦！
Abstract
Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action, a joint affordance localization and arm allocation problem that geometry-only planners cannot resolve without semantic understanding of task intent. Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometric heuristics for arm assignment, failing to jointly reason about task-relevant contact regions and arm allocation. We reframe bimanual manipulation as a joint affordance localization and arm allocation problem and propose a hierarchical framework for task-aware bimanual affordance prediction that leverages a Vision-Language Model (VLM) to generalize across object categories and task descriptions without requiring category-specific training. Our approach fuses multi-view RGB-D observations into a consistent 3D scene representation and generates global 6-DoF grasp candidates, which are then spatially and semantically filtered by querying the VLM for task-relevant affordance regions on each object, as well as for arm allocation to the individual objects, thereby ensuring geometric validity while respecting task semantics. We evaluate our method on a dual-arm platform across nine real-world manipulation tasks spanning four categories: parallel manipulation, coordinated stabilization, tool use, and human handover. Our approach achieves consistently higher task success rates than geometric and semantic baselines for task-oriented grasping, demonstrating that explicit semantic reasoning over affordances and arm allocation helps enable reliable bimanual manipulation in unstructured environments.