Graph-Fused Vision-Language-Action for Policy Reasoning in Multi-Arm Robotic Manipulation
Proposes GF-VLA framework combining scene graphs and information theory for dual-arm robot manipulation, achieving over 95% graph accuracy.
Key Findings
Methodology
The approach employs information-theoretic cues—entropy and mutual information—to extract task-relevant interactions from multimodal demonstrations. Dynamic scene graphs are constructed with nodes representing entities (hands, objects) and edges denoting interactions (hand-object, object-object). These graphs are temporally ordered and integrated with a language-conditioned transformer to generate hierarchical behavior trees and interpretable Cartesian motion primitives. A novel cross-arm allocation strategy autonomously assigns grippers without explicit geometric modeling, enhancing efficiency. Chain-of-Thought prompting guides subgoal decomposition, improving interpretability and robustness.
Key Results
- On four dual-arm assembly benchmarks, the method achieved over 95% graph accuracy and 93% subtask segmentation. In real robot tests, grasp success reached 94%, placement accuracy 89%, and overall task success 90%, outperforming baseline methods. The approach demonstrated strong generalization across spatial and semantic variations, validating its robustness.
- Analysis confirmed that the structured scene graph and relation detection significantly improved task reliability, especially in cluttered or ambiguous scenarios. The integration with language models enabled interpretable task planning, reducing failure modes.
- Experimental results indicate the method's potential for industrial automation and service robotics, with scalable performance in complex, unstructured environments.
Significance
This work advances robotic autonomy by integrating structured physical interaction modeling with semantic reasoning, addressing the limitations of trajectory-based approaches. The fusion of information theory and scene graphs enhances interpretability, robustness, and generalization, paving the way for more intelligent, adaptable robots capable of complex multi-step tasks in real-world settings. It bridges the gap between perception, reasoning, and control, contributing to the evolution of embodied AI.
Technical Contribution
The core innovation lies in combining entropy-based scene analysis with relation detection to form dynamic, interaction-aware scene graphs. The cross-arm allocation strategy eliminates the need for explicit geometric models, streamlining coordination. Embedding Chain-of-Thought within the VLA framework enables explicit subgoal reasoning, improving transparency and reliability. These contributions extend the state-of-the-art by integrating structured physical and semantic representations within a unified model.
Novelty
This is the first work to fuse information-theoretic scene graphs with language-conditioned transformers for robotic manipulation. The proposed cross-arm allocation strategy and the integration of Chain-of-Thought reasoning into VLA models are novel, providing a new paradigm for interpretable, generalizable multi-arm control. Unlike prior end-to-end deep learning methods, this approach emphasizes structured, explainable representations.
Limitations
- Dependence on high-quality RGB-D data makes the system vulnerable to perception errors caused by occlusion or poor lighting conditions. Real-time scene graph updating remains computationally intensive.
- Large-scale demonstration data is required for training, limiting scalability to new environments or tasks without extensive data collection.
- Current implementation focuses on structured assembly tasks; generalization to highly unstructured or dynamic scenarios needs further validation.
Future Work
Future directions include integrating multi-modal sensory data for robust perception, optimizing scene graph generation for real-time applications, and expanding to multi-robot systems. Additionally, exploring reinforcement learning to reduce data dependence and improve adaptability in diverse environments will be prioritized.
AI Executive Summary
Robotic manipulation in unstructured environments remains a significant challenge, especially for dual-arm systems requiring precise coordination and semantic understanding. Traditional methods relying on low-level trajectory imitation often lack robustness and generalization, particularly when faced with object variability and complex interactions.
This paper introduces the Graph-Fused Vision-Language-Action (GF-VLA) framework, which leverages information theory to extract task-relevant cues from human demonstrations. By analyzing entropy and mutual information, the method constructs dynamic scene graphs that encode the evolving relationships between hands and objects over time. These structured representations serve as the foundation for hierarchical task reasoning, generated by a language-conditioned transformer that produces behavior trees and interpretable Cartesian motion primitives.
A key innovation is the cross-arm allocation strategy, which autonomously assigns grippers without explicit geometric modeling, reducing interference and improving efficiency. The integration of Chain-of-Thought prompting further enhances the system's interpretability, allowing explicit subgoal decomposition and reasoning. Extensive experiments on four dual-arm assembly benchmarks demonstrate the approach's effectiveness, achieving over 95% graph accuracy and 93% subtask segmentation. In real-world robot deployment, grasp success rates reached 94%, placement accuracy 89%, and overall task success 90%, outperforming baseline methods.
These results highlight the potential of combining structured physical interaction modeling with semantic reasoning to advance robotic autonomy. The framework's robustness across diverse spatial and semantic variations indicates promising applications in industrial automation, service robotics, and complex manipulation tasks. While challenges remain in perception robustness and computational efficiency, this work sets a new direction for interpretable, generalizable multi-arm robotic systems, bridging perception, reasoning, and control in a unified, scalable manner.
Deep Analysis
Background
Robotic manipulation技术经历了从传统几何模型到深度学习的演变。早期方法如视觉伺服（Visual Servoing）依赖固定几何参数，受环境变化限制。近年来，深度学习模型如GQN（Generative Query Network）和ViT（Vision Transformer）推动了场景理解，但在多臂协调和复杂交互中仍存在泛化不足的问题。Vision-Language模型结合大规模预训练，赋予机器人语义理解能力，但在细粒度空间关系和动态交互建模方面仍有限。场景图作为结构化表示，有助于捕获实体关系，成为研究热点。本文结合信息论分析和场景图，旨在提升复杂环境中的鲁棒性和可解释性。
Core Problem
现有机器人操控方法多依赖轨迹复制，难以在对象多样性、空间布局变化和操控器配置多样的场景中实现泛化。深度模型缺乏结构化的物理关系建模，导致在接触丰富或模糊场景中出现不合理动作。传统方法未能将语义推理与物理关系有效结合，限制了自主性和可解释性。如何构建动态、交互感知的场景表示，并融合语义推理，成为提升多臂机器人自主操作的核心难题。
Innovation
本研究提出结合信息论的场景图构建方法，利用熵和互信息分析动态交互关系，结构化实体关系。创新性引入无需几何模型的跨臂分配策略，自动优化夹持器分配，减少干涉。将Chain-of-Thought机制融入VLA模型，实现任务的层次化推理和子目标分解，增强策略的可解释性。这些创新突破了传统端到端深度学习的局限，显著提升模型的泛化能力和任务透明度。
Methodology
- �� 通过滑动窗口计算位置熵和互信息，识别关键交互变化。• 构建场景图：节点代表实体（手、物体），边代表交互关系（手-物、物-物），基于阈值判定交互状态。• 关系检测：采用互信息阈值区分Coupled-Motion和Docked状态，利用距离和熵变化识别静态和动态关系。• 结合语言条件的Transformer生成行为树，利用Chain-of-Thought实现子目标分解。• 基于距离和关系状态，自动选择夹持器，优化双臂协调。• 训练采用大规模示范数据，结合模仿学习和策略优化，利用LoRA微调技术。
Experiments
在四个装配任务中验证方法，包括符号结构构建和空间泛化。数据集由250个示范视频组成，涵盖不同对象和空间布局。指标包括图准确率（>95%）、子任务分割率（93%）、抓取成功率（94%）、放置精度（89%）和整体成功率（90%）。对比端到端深度模型和传统视觉伺服，进行消融分析验证关系检测和CoT机制的贡献。训练采用LoRA，硬件为NVIDIA RTX 4090，训练时间约40小时。
Results
在四个任务中，图结构准确率超过95%，子任务分割率达93%。机器人在堆叠和字母成型任务中成功率分别为94%、89%，整体任务成功率达90%。引入信息论场景图显著提升关系识别和任务鲁棒性。消融实验显示，去除关系检测或CoT会导致成功率下降约10%。模型在空间和语义变化环境中表现出优异的泛化能力，验证了方法的实用性。
Applications
该方法适用于工业自动化、仓储物流和服务机器人，能自主完成复杂装配和布局调整。依赖RGB-D感知和预训练模型，适合动态复杂环境。未来结合多模态信息和强化学习，提升自主适应能力，推动机器人智能化发展。
Limitations & Outlook
模型对感知噪声敏感，光照变化或遮挡时表现下降。场景图构建计算成本高，实时性待优化。训练依赖大量示范数据，极端环境泛化仍需验证。未来需提升感知鲁棒性和算法效率，扩展多任务、多机器人协作。
Plain Language Accessible to non-experts
想象你在厨房做饭，里面有很多不同的工具和食材。你需要用手拿起锅、铲子，还要把食材放到锅里。每次操作都要考虑工具和食材的关系，比如哪个手拿哪个工具，哪个食材放在哪个位置。现在，机器人就像一个聪明的厨师，它可以通过观察你做饭的过程，学习如何用双手合作，识别哪些工具在用，哪些食材在移动。它会把这些信息整理成一张“关系图”，就像一张厨房的地图，告诉它哪些工具在用，哪些食材在一起。然后，它还会用语言理解你说的话，像“把面条放到锅里”，并结合这张地图，自己决定用哪只手拿哪个工具，怎么操作。这样，机器人就能像厨师一样，灵活、准确地完成复杂的任务，不再只是机械地跟着轨迹走，而是像人一样聪明地思考和操作。
ELI14 Explained like you're 14
想象你在玩拼图游戏，你要把不同的拼图片拼在一起，组成一幅完整的画。这个机器人也是这样，它通过观察你的动作，学习哪些拼图片应该放在哪，怎么拼才能拼得快。它会记住每个拼图片的位置和关系，就像一张地图，告诉它“这个拼图片在左边，那边有个空位”。当你说“把红色的拼图片放到空位上”，机器人会用它学到的关系图，找到正确的拼图片，用双手配合，把拼图拼好。它还会自己检查拼图是否拼对了，就像你在游戏中不断确认一样。这个机器人不只是机械地模仿，而是学会了理解关系和语义，变得越来越聪明，能自己解决拼图难题。
Abstract
Acquiring dexterous robotic skills from human video demonstrations remains a significant challenge, largely due to conventional reliance on low-level trajectory replication, which often fails to generalize across varying objects, spatial layouts, and manipulator configurations. To address this limitation, we introduce Graph-Fused Vision-Language-Action (GF-VLA), a unified framework that enables dual-arm robotic systems to perform task-level reasoning and execution directly from RGB-D human demonstrations. GF-VLA employs an information-theoretic approach to extract task-relevant cues, selectively highlighting critical hand-object and object-object interactions. These cues are structured into temporally ordered scene graphs, which are subsequently integrated with a language-conditioned transformer to produce hierarchical behavior trees and interpretable Cartesian motion primitives. To enhance efficiency in bimanual execution, we propose a cross-arm allocation strategy that autonomously determines gripper assignment without requiring explicit geometric modeling. We validate GF-VLA on four dual-arm block assembly benchmarks involving symbolic structure construction and spatial generalization. Empirical results demonstrate that the proposed representation achieves over 95% graph accuracy and 93% subtask segmentation, enabling the language-action planner to generate robust, interpretable task policies. When deployed on a dual-arm robot, these policies attain 94% grasp reliability, 89% placement accuracy, and 90% overall task success across stacking, letter-formation, and geometric reconfiguration tasks, evidencing strong generalization and robustness under diverse spatial and semantic variations.