CUBic: Coordinated Unified Bimanual Perception and Control Framework
CUBic employs shared codebooks for unified bimanual perception and control, achieving 12% higher success in RoboTwin benchmarks.
Key Findings
Methodology
CUBic integrates three core modules: unidirectional perception aggregation using masked attention to fuse multi-view data, bidirectional perception coordination via shared codebooks enabling implicit arm interaction, and a unified diffusion-based policy translating perceptual tokens into synchronized dual-arm actions. The architecture promotes intrinsic coordination by structuring perception and control within a shared latent space, trained end-to-end with a two-stage process—initial independent perception learning followed by frozen perception and joint action refinement. Specific algorithms include DiT-based diffusion models, vector quantization (VQ), and multi-view attention mechanisms, collectively ensuring robust, physically consistent bimanual manipulation.
Key Results
- On the RoboTwin benchmark, CUBic achieved an average success rate of 51.8%, outperforming baseline methods like Diffusion Policy (38.5%) and DP3 (39.8%). In complex tasks such as dual bottle stacking and apple grasping, success rates increased by approximately 10-15%. Ablation studies confirmed the importance of shared codebooks and perception coordination modules, with performance drops observed when these components were removed. The model demonstrated strong generalization across simulated and real-world scenarios, maintaining high coordination accuracy and task success.
- In single-arm tasks, CUBic improved localization and grasp success, indicating effective perception fusion. In dual-arm tasks, it reduced conflicts and improved synchronization, showcasing the benefit of the shared latent space. The experiments validated the architecture's ability to balance independence and interaction, leading to more natural and reliable bimanual control.
- The results highlight the potential of unified perceptual representations combined with diffusion-based control, setting new state-of-the-art benchmarks and opening avenues for scalable, flexible multi-arm robotic systems.
Significance
This work addresses a fundamental challenge in robotics: how to achieve seamless coordination between multiple manipulators using visual inputs. By unifying perception and control within a shared latent space, CUBic overcomes the limitations of previous decoupled or strongly coupled frameworks. The approach enhances both the autonomy and robustness of multi-arm robots, enabling more complex, precise, and adaptable behaviors. Its implications extend to industrial automation, collaborative robots, and autonomous systems, where flexible, reliable multi-manipulator operation is critical. The integration of diffusion models and shared encoding mechanisms marks a significant step toward more intelligent, scalable robotic systems capable of complex collaborative tasks.
Technical Contribution
CUBic introduces a novel shared codebook-based representation for multi-view perception and control, enabling implicit coordination without explicit coupling. It combines multi-view masked attention for local-global perception fusion, dual codebooks for latent interaction, and a diffusion transformer for action generation, all trained end-to-end. This architecture bridges perception and control seamlessly, providing a flexible, interpretable, and scalable framework for bimanual manipulation. The use of hierarchical training stages ensures stability and generalization, setting a new paradigm for multi-modal, multi-arm robotic learning.
Novelty
This is the first work to model dual-arm perception and control within a unified shared latent space via dual codebooks, enabling implicit coordination. Unlike prior methods relying on explicit role assignment or structural constraints, CUBic’s architecture allows natural emergence of cooperation from structured representations, significantly advancing multi-arm robotic learning and perception fusion. Its integration of multi-view attention, vector quantization, and diffusion models offers a comprehensive, innovative solution to longstanding challenges in bimanual manipulation.
Limitations
- The model's robustness in highly dynamic, cluttered, or occluded environments remains limited, requiring further enhancement of perception modules.
- Training demands large datasets and computational resources, which may hinder real-world deployment without optimization.
- Current experiments are primarily in controlled settings; performance in diverse, unstructured scenarios needs further validation.
Future Work
Future research will focus on improving perception robustness under occlusion and dynamic conditions, integrating reinforcement learning for adaptive control, and reducing computational costs. Expanding the framework to more complex multi-robot systems and real-world industrial applications will be key, aiming for autonomous, scalable, and versatile multi-arm robots capable of complex collaborative tasks in unstructured environments.
AI Executive Summary
The rapid advancement of visuomotor policies has transformed robotic control, yet multi-arm coordination remains a significant challenge. Existing approaches often treat perception and control separately or enforce rigid coupling, limiting flexibility and robustness. Addressing this, the CUBic framework introduces a unified, shared latent space for bimanual perception and control, leveraging a novel dual codebook mechanism. This design allows the model to naturally balance independence and coordination, enabling synchronized actions without explicit coupling.
CUBic’s architecture comprises three key modules: unidirectional perception aggregation, which fuses multi-view sensory inputs via masked attention; bidirectional perception coordination, which employs shared codebooks to facilitate implicit inter-arm communication; and a diffusion-based policy that maps perceptual tokens to synchronized actions. The entire system is trained end-to-end through a hierarchical process—initial independent perception learning followed by joint action refinement—ensuring stability and generalization.
Experimental results on the RoboTwin benchmark demonstrate that CUBic surpasses state-of-the-art methods, achieving an average success rate of 51.8%, a 12% improvement over previous best. It excels in complex tasks like dual bottle stacking and apple grasping, showing robustness and adaptability in both simulated and real-world scenarios. These findings highlight the potential of unified perceptual representations combined with diffusion models for scalable, flexible multi-arm robotic control.
This work marks a significant step toward autonomous, collaborative robots capable of complex manipulation tasks. By unifying perception and control within a shared latent space, CUBic opens new avenues for industrial automation, service robots, and human-robot collaboration. Despite current limitations in dynamic environments and computational costs, ongoing research aims to enhance robustness, efficiency, and scalability, bringing us closer to truly intelligent multi-arm robotic systems.
Deep Dive
Plain Language Accessible to non-experts
想象一个厨房里有两个厨师同时做菜。每个厨师都需要知道自己手里的材料和位置，同时还要知道另一个厨师的动作，以免碰撞或重复工作。传统的方法要么让两个厨师完全独立，各自做事，要么让他们紧密合作，彼此依赖。CUBic就像给两个厨师配备了一个共同的菜单和秘密信号系统，让他们既能自己专注，又能默契配合。它用一种特殊的“语言”让两个厨师在不打扰对方的情况下，知道彼此的动作和位置，从而一起高效完成菜肴。这种方法让厨房里的合作变得更自然、更顺畅，也更智能。
ELI14 Explained like you're 14
想象你和朋友在玩一款合作游戏。你们都要完成任务，但不能一直盯着对方看，否则会打扰。你们需要一种秘密的信号或暗示，让对方知道你的动作，同时自己也能理解对方在做什么。CUBic就像给机器人配备了这种秘密信号系统，让两个机械手可以同时做事，又不会撞到一起。它用一种特别的“语言”把两个机械手的动作和感知信息放在一起，让它们既能自己做事，又能配合得很好。这样，机器人就能像两个默契的厨师一样，合作完成复杂的任务，不仅效率高，还很稳当。
Abstract
Recent advances in visuomotor policy learning have enabled robots to perform control directly from visual inputs. Yet, extending such end-to-end learning from single-arm to bimanual manipulation remains challenging due to the need for both independent perception and coordinated interaction between arms. Existing methods typically favor one side -- either decoupling the two arms to avoid interference or enforcing strong cross-arm coupling for coordination -- thus lacking a unified treatment. We propose CUBic, a Coordinated and Unified framework for Bimanual perception and control that reformulates bimanual coordination as a unified perceptual modeling problem. CUBic learns a shared tokenized representation bridging perception and control, where independence and coordination emerge intrinsically from structure rather than from hand-crafted coupling. Our approach integrates three components: unidirectional perception aggregation, bidirectional perception coordination through two codebooks with shared mapping, and a unified perception-to-control diffusion policy. Extensive experiments on the RoboTwin benchmark show that CUBic consistently surpasses standard baselines, achieving marked improvements in coordination accuracy and task success rates over state-of-the-art visuomotor baselines.