Polybot: Training One Policy Across Robots While Embracing Variability
Polybot aligns observation, action, and internal features across robots via wrist cameras, inverse kinematics, contrastive learning, and multi-head policies, enabling zero-shot transfer and improved success rates.
Key Findings
Methodology
This work introduces a hierarchical alignment framework combining wrist-mounted visual observations, shared inverse kinematics, and contrastive learning to unify feature representations across diverse robots. First, wrist cameras capture consistent end-effector views, reducing visual variability. Second, a shared inverse kinematics solver aligns pose commands, while robot-specific controllers handle low-level execution. Third, contrastive learning pretrains feature encoders to map similar states across robots, facilitating internal representation alignment. Finally, a multi-headed policy architecture enables robot-specific dynamics modeling. The approach is validated on a dataset of 60 hours across 6 tasks and 3 robots (WidowX 250S, Franka Panda, Sawyer), demonstrating significant improvements in success rate and sample efficiency, especially in zero-shot and few-shot scenarios.
Key Results
- Polybot achieves over 70% success rate on new tasks with minimal data, outperforming single-robot baselines by more than 19%. Zero-shot transfer on structurally similar tasks reaches 40-60% success, a substantial leap over prior methods. The multi-headed policy with contrastive pretraining outperforms ablations lacking these components, especially on complex 6-DoF tasks, with success rates exceeding 90%.
- Experiments show that unaligned models drop success by 19%, highlighting the importance of internal feature alignment. Wrist camera-based observation space reduces visual domain gap, enabling robust generalization. The shared inverse kinematics ensures pose consistency, while robot-specific heads capture unique dynamics, collectively enabling effective cross-robot transfer.
- In multi-task and multi-environment tests, Polybot demonstrates superior performance in shelf manipulation and object placement tasks, with success rates surpassing 85%. The framework's ability to leverage prior data accelerates learning and reduces data collection costs, advancing practical deployment of vision-based robotic systems.
Significance
This research addresses a fundamental challenge in robotic learning: how to efficiently reuse large datasets across heterogeneous platforms. By systematically aligning observation, action, and internal representations, it unlocks the potential for scalable, generalizable robotic policies. This breakthrough reduces the need for extensive data collection on each robot, lowering costs and enabling rapid deployment in industrial, service, and collaborative settings. The approach also paves the way for more adaptive, multi-robot systems capable of learning new tasks with minimal supervision, significantly advancing the field of autonomous manipulation.
Technical Contribution
The paper introduces a multi-level alignment strategy that combines spatial standardization via wrist cameras, a shared inverse kinematics module, and contrastive learning for feature alignment. The multi-headed policy architecture allows for robot-specific dynamics modeling, overcoming the limitations of fixed action spaces. These innovations collectively enable effective transfer across diverse robotic platforms, outperforming existing domain adaptation and transfer learning methods. The framework also provides theoretical insights into internal representation alignment, contributing to the broader understanding of multi-robot generalization.
Novelty
This work is the first to integrate multi-level spatial and feature alignment with a multi-headed policy architecture for cross-robot transfer. Unlike prior approaches limited to fixed viewpoints or low-dimensional observations, it handles high-dimensional visual inputs and complex 6-DoF tasks. The combination of wrist-mounted visual space standardization, contrastive feature alignment, and robot-specific dynamics modeling represents a novel paradigm in robotic transfer learning, setting new benchmarks in success rates and sample efficiency.
Limitations
- The reliance on wrist-mounted cameras assumes consistent end-effector views, which may be disrupted by occlusions or extreme poses. The approach may struggle in highly cluttered or dynamic environments where visual features are ambiguous.
- Training complexity and computational costs are high due to multi-component pretraining and fine-tuning, limiting real-time applicability. Extending to more diverse hardware configurations requires additional adaptation.
- While effective for similar tasks, the framework's performance on highly dissimilar or novel tasks remains to be thoroughly tested. Further research is needed to improve robustness and scalability.
Future Work
Future directions include enhancing visual robustness through multi-view or 3D perception, integrating reinforcement learning for dynamic policy adaptation, and expanding to multi-agent collaborative scenarios. Developing more efficient training algorithms and exploring unsupervised domain adaptation techniques could further reduce data requirements. Additionally, extending the framework to handle more diverse hardware and environmental variations will be crucial for real-world deployment.
AI Executive Summary
Robotic manipulation has long been hindered by the challenge of transferring learned skills across different hardware platforms. Traditional methods often require extensive retraining or data collection tailored to each robot, limiting scalability and practical deployment. Recognizing this bottleneck, the authors propose Polybot, a comprehensive framework designed to enable a single policy to operate effectively across multiple robots with minimal additional data.
At the core of Polybot is a multi-layered alignment strategy. The first layer leverages wrist-mounted cameras to standardize observation spaces, significantly reducing visual variability caused by differing viewpoints. The second layer employs a shared inverse kinematics solver to unify the action space, translating high-level commands into robot-specific joint movements. The third layer introduces contrastive learning to align internal feature representations, ensuring that the policy captures task-relevant states consistently across robots. Finally, a multi-headed policy architecture allows each robot to learn its unique dynamics while sharing common features.
Extensive experiments on a dataset spanning 60 hours, involving 6 tasks and 3 diverse robots, demonstrate the effectiveness of this approach. Results show success rates exceeding 70% on new tasks with minimal data, outperforming baselines by over 19%. Zero-shot transfer achieves 40-60% success in structurally similar tasks, highlighting the framework’s generalization capacity. The ablation studies confirm that each component—space alignment, contrastive pretraining, and multi-head design—contributes significantly to performance.
This work marks a significant advance in robotic learning, reducing data collection costs and enabling rapid adaptation to new tasks and platforms. Its implications extend to industrial automation, service robots, and multi-agent systems, where scalable, flexible manipulation policies are essential. Despite current limitations, such as visual occlusion sensitivity and training complexity, the proposed framework opens promising avenues for future research in autonomous, multi-robot collaboration and lifelong learning in robotics.
Deep Dive
Abstract
Reusing large datasets is crucial to scale vision-based robotic manipulators to everyday scenarios due to the high cost of collecting robotic datasets. However, robotic platforms possess varying control schemes, camera viewpoints, kinematic configurations, and end-effector morphologies, posing significant challenges when transferring manipulation skills from one platform to another. To tackle this problem, we propose a set of key design decisions to train a single policy for deployment on multiple robotic platforms. Our framework first aligns the observation and action spaces of our policy across embodiments via utilizing wrist cameras and a unified, but modular codebase. To bridge the remaining domain shift, we align our policy's internal representations across embodiments through contrastive learning. We evaluate our method on a dataset collected over 60 hours spanning 6 tasks and 3 robots with varying joint configurations and sizes: the WidowX 250S, the Franka Emika Panda, and the Sawyer. Our results demonstrate significant improvements in success rate and sample efficiency for our policy when using new task data collected on a different robot, validating our proposed design decisions. More details and videos can be found on our anonymized project website: https://sites.google.com/view/polybot-multirobot