Point Bridge: 3D Representations for Cross Domain Policy Learning
Point Bridge leverages unified point cloud representations for zero-shot sim-to-real transfer, achieving up to 44% performance gains.
Key Findings
Methodology
The approach integrates Vision-Language Models (VLMs) for automated scene keypoint extraction, employs a transformer-based policy network, and utilizes synthetic data augmentation. Scene filtering via VLM reduces visual domain gap, enabling robust cross-domain transfer. The pipeline includes point cloud generation from object meshes in simulation and VLM-guided keypoint detection in real scenes. Multi-task training with synthetic data, combined with limited real demonstrations, enhances generalization. The system supports real-time scene filtering, multi-sensor fusion, and policy inference, facilitating zero-shot transfer and fine-tuning.
Key Results
- On six real-world tasks, Point Bridge achieved up to 44% improvement in zero-shot sim-to-real transfer, with success rates rising from approximately 30% to over 70%. When co-trained with a small set of real demonstrations, performance further increased by 66%. Across different sensing modalities, the method maintained robustness, demonstrating effective domain invariance. Ablation studies confirmed that VLM-guided keypoint extraction and point cloud filtering are critical for performance gains, especially under occlusion and lighting variations.
- In multi-task settings, Point Bridge outperformed baselines, with success rates exceeding 80% in several tasks. The model showed strong adaptability to diverse object types and environmental conditions. Combining synthetic data with limited real demonstrations yielded significant improvements over purely simulation-trained policies. The results validate the effectiveness of the unified point cloud representation and multi-modal filtering in bridging the sim-to-real gap.
- Ablation experiments revealed that removing VLM scene filtering reduced transfer success by about 20%. Incorporating multi-view triangulation and sensor fusion strategies further improved robustness. The approach's scalability to multiple tasks and environments highlights its potential for practical deployment in industrial and service robots, reducing reliance on extensive real-world data collection.
Significance
This work addresses a fundamental challenge in robotics: enabling models trained in simulation to perform reliably in real environments without extensive manual alignment or data annotation. By introducing a domain-agnostic point cloud representation guided by vision-language models, it significantly reduces the dependency on costly real-world datasets. The framework's ability to leverage synthetic data for large-scale training and adapt with minimal real demonstrations paves the way for scalable, generalist robotic agents. Its implications extend to industrial automation, autonomous logistics, and service robotics, where rapid deployment and robust transfer are critical. The method's simplicity and effectiveness mark a substantial step toward practical, versatile robotic systems capable of learning from diverse data sources.
Technical Contribution
Point Bridge innovatively combines automated point cloud extraction via VLMs with a transformer-based policy architecture, enabling zero-shot transfer without explicit visual or object-level alignment. The key technical advance lies in the unified, domain-agnostic scene representation—point clouds—that bridge the simulation-reality gap. The system supports large-scale synthetic data augmentation, multi-task learning, and real-time scene filtering, significantly reducing the need for manual annotation and domain calibration. Its modular design allows easy integration with various sensors and environments, providing a scalable solution for policy transfer. The approach also introduces a novel pipeline for multi-modal scene filtering and keypoint detection, setting new standards for robustness and generalization in robotic manipulation.
Novelty
This is the first work to utilize Vision-Language Models for automated, domain-agnostic point cloud extraction in the context of sim-to-real transfer. Unlike prior methods relying on explicit object detection or manual annotations, Point Bridge leverages VLMs to identify task-relevant objects and extract keypoints without alignment. Its core innovation is representing scenes as unified point clouds, enabling zero-shot transfer across diverse environments and object types. The integration of synthetic data augmentation with minimal real data fine-tuning further distinguishes this approach, setting a new benchmark for scalable, generalist robotic policy learning.
Limitations
- The method's performance can degrade in highly cluttered or occluded scenes where keypoint detection becomes unreliable. Dependence on VLM accuracy limits robustness under extreme lighting or visual noise. Multi-task generalization still faces challenges in highly complex or novel tasks. Real-time inference speed needs optimization for deployment in fast-paced industrial settings. Additionally, the current framework assumes calibrated sensors and known camera parameters, which may not always be feasible in unstructured environments. Future work should address these issues to enhance robustness and scalability.
Future Work
Future directions include integrating self-supervised learning to improve keypoint detection robustness, exploring adaptive sensor fusion techniques, and optimizing inference pipelines for real-time deployment. Extending the framework to handle unstructured or dynamic environments, incorporating active perception strategies, and scaling to more complex multi-agent scenarios are also promising avenues. Additionally, developing end-to-end learning systems that jointly optimize perception and policy could further enhance transfer performance. These advancements aim to realize fully autonomous, adaptable robots capable of operating seamlessly across diverse real-world settings.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在学校里学会了一种新游戏规则，但每次换到不同的学校或不同的游戏场地，你都需要重新学习。这个方法就像教机器人用一种特殊的“点点地图”来记住场景里的重要位置，比如桌子、椅子、门的位置。只要它学会了这些关键点，就能在新环境中快速找到目标，不用每次都重新学习全部细节。它还用一种叫“超级大脑”的算法，把这些点点组合成一幅场景图，让机器人知道怎么操作。这样，无论环境怎么变，机器人都能用这些“点点地图”完成任务，就像我们用简洁的地图导航一样。这个技术让机器人变得更聪明、更灵活，能在不同的房间、不同的场景中工作得更好。
ELI14 Explained like you're 14
想象你在玩一款超级酷的游戏，但每个关卡都不一样，你得重新学怎么操作。这个研究就像教机器人玩这个游戏一样，它用一种特别的方法，把场景变成一堆点点，就像星星一样，然后用聪明的算法把这些点点拼成一张地图。这样，无论场景怎么变，机器人都能看懂，知道怎么做事。它还用一种叫Transformer的“超级大脑”来学习怎么操作，像我们用大脑记攻略一样。实验发现，这个方法让机器人在新场景里表现得比以前好多了，成功率提高了不少。未来，这样的机器人可以帮我们做很多事情，比如在工厂搬东西，或者在家帮忙打扫。虽然还不完美，但已经走出了很大一步。
Abstract
Robot foundation models are beginning to deliver on the promise of generalist robotic agents, yet progress remains constrained by the scarcity of large-scale real-world manipulation datasets. Simulation and synthetic data generation offer a scalable alternative, but their usefulness is limited by the visual domain gap between simulation and reality. In this work, we present Point Bridge, a framework that leverages unified, domain-agnostic point-based representations to unlock synthetic datasets for zero-shot sim-to-real policy transfer, without explicit visual or object-level alignment. Point Bridge combines automated point-based representation extraction via Vision-Language Models (VLMs), transformer-based policy learning, and efficient inference-time pipelines to train capable real-world manipulation agents using only synthetic data. With additional co-training on small sets of real demonstrations, Point Bridge further improves performance, substantially outperforming prior vision-based sim-and-real co-training methods. It achieves up to 44% gains in zero-shot sim-to-real transfer and up to 66% with limited real data across both single-task and multitask settings. Videos of the robot are best viewed at: https://pointbridge3d.github.io/