BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation
BridgeVLA++ integrates multi-view projection, explicit spatio-temporal memory, and pre-trained VLMs for efficient, robust 3D robot manipulation.
Key Findings
Methodology
The proposed BridgeVLA++ framework employs multi-view orthographic projections of point clouds to generate 2D spatial heatmaps, leveraging pre-trained vision-language models (VLMs) such as SigLIP and Gemma for semantic spatial localization. The core innovation is the integration of a unified spatio-temporal memory architecture that explicitly stores historical observations and geometric information, enabling reasoning over observation sequences. During pre-training, the model learns to generate language-conditioned heatmaps from large-scale object grounding datasets like RoboPoint, optimizing cross-entropy loss for spatial localization. In downstream tasks, point clouds are rendered into multiple views, processed by the VLM to produce heatmaps, which are then back-projected to recover 3D target positions. The memory modules maintain keyframes and interaction history, facilitating occlusion recovery and long-term reasoning. The entire pipeline supports end-to-end training and transfer to real-world robotic platforms, demonstrating high data efficiency and generalization across tasks.
Key Results
- On five simulation benchmarks including RLBench, COLOSSEUM, and GemBench, BridgeVLA achieved an average success rate of 93.8%, outperforming prior methods like Lift3D and PointVLA by 10-15%. In memory-dependent tasks such as RMBench and MemoryBench, the model's success rate exceeded 92%, showing significant improvements. In real-world experiments with Franka Research 3 and Dobot CR5A robots, the model trained with fewer than 10 demonstrations achieved success rates of 93.3%, a stark contrast to the 20% success rate of baseline models. Ablation studies confirmed that the explicit spatio-temporal memory contributed approximately 15-20% performance gain, especially under occlusion and dynamic scene conditions. The model maintained high robustness to environmental variations, sensor noise, and unseen objects, demonstrating excellent transferability.
- Furthermore, the integration of multi-view projections and memory modules enabled the model to recover occluded geometry and maintain task context over long sequences, significantly improving manipulation precision and success in complex scenarios. Quantitative metrics such as spatial localization error decreased by 25%, and success rates in multi-step tasks increased by 20% compared to memory-free versions. These results validate the effectiveness of the architecture in both simulated and real environments, highlighting its potential for scalable, data-efficient robotic manipulation.
Significance
This work addresses fundamental challenges in robotic manipulation: data efficiency, generalization, and memory-dependent reasoning. By unifying semantic understanding from pre-trained VLMs with geometric reasoning via multi-view projections and explicit memory, the framework overcomes the limitations of prior approaches that either lack memory or require extensive data. The ability to perform robustly in open-world, occluded, and dynamic environments marks a significant step toward autonomous robots capable of complex, long-horizon tasks. Such advancements have broad implications for industrial automation, service robotics, and autonomous exploration, where data collection is costly and environments unpredictable. The framework's scalability across tasks, environments, and robotic platforms further underscores its potential for real-world deployment, reducing reliance on task-specific engineering and extensive retraining.
Technical Contribution
The paper's key technical contributions include: 1) a multi-view orthographic projection mechanism that preserves geometric consistency and facilitates the use of pre-trained VLMs for spatial localization; 2) the design of a unified spatio-temporal memory architecture that explicitly stores and reasons over historical observations, enabling occlusion recovery and long-term planning; 3) a transfer learning pipeline that aligns 2D heatmap prediction with 3D manipulation tasks, leveraging large-scale object grounding datasets for pre-training; 4) an end-to-end training framework that maintains data efficiency while supporting complex multi-task and multi-robot scenarios. This approach bridges semantic language understanding with geometric spatial reasoning, opening new avenues for scalable, memory-aware robotic manipulation.
Novelty
This work is the first to integrate pre-trained vision-language models with explicit multi-view spatial heatmap prediction and a unified spatio-temporal memory architecture for 3D robot manipulation. Unlike prior models such as Lift3D or PointVLA, which either rely on specialized 3D encoders or lack explicit memory, BridgeVLA++ maintains a simple yet powerful pipeline grounded in 2D visual-semantic space. Its novel combination of multi-view projection, language-conditioned heatmaps, and explicit memory modules enables robust reasoning over long observation sequences and occluded scenes, significantly advancing the state-of-the-art in data-efficient, generalizable 3D manipulation.
Limitations
- Despite its strengths, the model's performance can degrade in highly dynamic or cluttered environments with severe occlusion, where geometric recovery becomes challenging. The reliance on high-quality point cloud rendering and multi-view consistency may be affected by sensor noise or environmental factors, limiting robustness in real-world deployment. Additionally, the current architecture, while scalable, still requires substantial computational resources for training and inference, which could hinder real-time applications on resource-constrained robots. Future work should focus on optimizing efficiency, enhancing robustness to environmental variability, and extending capabilities to dynamic scene understanding.
Future Work
未来的研究将集中在：一是结合多模态感知（如触觉、力觉）以增强操作的细腻度和鲁棒性；二是探索更高效的模型结构，降低计算成本，实现边缘设备上的实时部署；三是扩展多机器人协作能力，通过共享记忆和空间推理，实现复杂任务的协同完成。此外，还将研究模型在极端环境和动态场景中的适应性，推动机器人自主学习和持续适应能力的发展。
AI Executive Summary
在机器人自主操控领域，空间理解和记忆能力一直是核心难题。传统方法多依赖大量标注数据，且在复杂环境中泛化能力有限。近年来，预训练的视觉语言模型（VLM）为机器人赋予了强大的语义理解能力，但其在三维空间中的应用仍面临几何信息不足和记忆缺失的问题。为解决这一难题，本文提出了BridgeVLA++，一种融合多视角投影、显式时空记忆的统一框架。
该框架的核心思想是：将3D场景通过多视角正交投影转化为二维空间中的热图，利用预训练的VLM进行空间目标定位。通过引入时空记忆模块，模型能够存储和推理观察历史，恢复被遮挡的几何信息，增强对复杂场景的理解能力。在训练阶段，模型在大规模对象定位数据（如RoboPoint）上学习空间热图生成，然后迁移到机器人操控任务中，利用多视角渲染和反投影实现精确的3D目标定位。
实验结果显示，BridgeVLA++在五个模拟基准（如RLBench、COLOSSEUM）中均实现了优异表现，成功率提升至93%以上，显著优于现有方法。在真实机器人平台（如Franka Research 3）上，模型在少样本条件下，成功率从20%跃升至93.3%，验证了其在实际应用中的潜力。这一技术突破不仅提升了机器人在复杂环境中的操作能力，也为未来自主学习、开放任务提供了坚实基础。
整体而言，BridgeVLA++通过结合语义理解与几何推理，开启了机器人空间认知的新篇章。其在数据效率、泛化能力和记忆推理方面的创新，为机器人自主性和智能化发展提供了重要技术支撑。未来，随着模型的不断优化和多模态信息的融合，其在工业、服务和探索等多个领域的应用前景将更加广阔。
Deep Analysis
Background
机器人自主操控技术经历了从传统特征工程到深度学习的快速演进。早期方法依赖手工设计的特征和规则，难以应对复杂环境中的多变场景。随着深度卷积网络（CNN）和变换器（Transformer）的出现，视觉感知和动作决策能力得到了极大提升。Lift3D、PointVLA等模型通过点云处理和多视角投影，增强了空间几何理解能力，同时结合预训练模型（如DINO、CLIP）实现语义理解。尽管如此，现有模型在遮挡、长序列依赖和多任务场景中表现仍有限，特别是在实际复杂环境中的鲁棒性不足。
Core Problem
核心问题在于如何结合预训练VLM的语义理解能力与空间几何信息，提升机器人在复杂环境中的操作精度和泛化能力。传统方法多依赖大量标注数据，且在遮挡和长时序任务中表现不佳。现有3D VLA模型缺乏显式的记忆机制，难以存储和推理观察历史，导致在多步骤任务中表现不稳定。此外，模型在实际应用中面临传感器噪声、遮挡和环境变化的挑战，如何设计高效、鲁棒的空间推理与记忆机制成为亟待解决的问题。
Innovation
本研究的创新点主要体现在三个方面：第一，提出基于多视角正交投影的空间热图预测机制，结合预训练的VLM实现空间目标定位，保持输入输出空间的一致性；第二，设计了统一的时空记忆架构，显式存储观察历史和几何信息，支持长序列推理和遮挡恢复；第三，将预训练模型迁移到3D操控任务中，采用多视角渲染与反投影技术，有效融合语义与几何信息，提升操作精度。该架构无需专门设计3D编码器，简洁高效，兼具泛化性和扩展性。
Methodology
- �� 预训练阶段：利用大规模对象定位数据（如RoboPoint）训练VLM生成空间热图，学习空间目标定位和语义空间关系，采用高斯概率图作为目标，优化交叉熵损失。
- �� 微调阶段：将点云渲染为多视角正交投影图像，输入预训练模型，预测空间热图。通过反投影恢复目标的3D空间位置。
- �� 时空记忆：引入时间记忆模块，存储关键帧和交互历史信息，支持长序列推理；空间记忆模块用于恢复遮挡几何信息，增强空间理解。
- �� 目标定位：在多视角热图基础上，采样候选点，计算热图值的加权和，选择最优目标位置。
- �� 旋转与操作：结合全局特征和局部特征，利用MLP预测目标旋转、夹爪状态和避障信息。
- �� 粗到细策略：先在全局范围内粗略定位目标，再在局部区域内进行细粒度调整，提升操作精度。训练中加入随机变换增强模型鲁棒性。
Experiments
在RLBench、COLOSSEUM、GemBench、RMBench和MemoryBench五个模拟环境中，模型采用成功率和空间定位误差作为主要指标，进行多轮对比。训练数据包括少量示范（少于10次）和大规模对象定位数据，超参数经过调优。真实机器人平台（Franka Research 3、Dobot CR5A）上，模型在少样本条件下完成多任务操作，验证迁移能力。对比无记忆版本和不同记忆策略，验证记忆模块的贡献。Ablation研究显示，加入时空记忆后，成功率提升15-20%，模型在遮挡和动态场景中的表现更稳健。
Results
在模拟环境中，BridgeVLA实现了平均成功率93.8%，在记忆依赖任务如RMBench中成功率超过92%，优于Lift3D和PointVLA。真实机器人平台上，少样本训练条件下，成功率从20%提升至93.3%，验证其极强的样本效率。模型在复杂场景中的空间推理能力得到验证，遮挡和动态变化条件下表现优异。定量指标如空间定位误差降低了25%，多步骤任务成功率提升了20%。这些结果证明了模型在模拟和真实环境中的优越性能，显示其广泛的应用潜力。
Applications
该技术适用于工业自动化中的装配、仓储机器人、服务机器人中的物品抓取与交互，以及自主探索任务。其优势在于少样本学习和强泛化能力，适合在数据有限或环境复杂的场景中部署。未来可结合多模态感知（如触觉、力觉）实现更精细的操控，推动机器人在未知环境中的自主适应与学习。
Limitations & Outlook
模型在极端遮挡和动态场景中仍存在几何恢复不足的问题，长时间序列中记忆的持续性和准确性有待提升。传感器噪声和环境变化可能影响多视角渲染效果，增加鲁棒性难度。训练和推理依赖大量计算资源，未来需优化模型结构以实现更高的实时性和能效。
Plain Language Accessible to non-experts
想象你在厨房里做饭。你需要记住每个材料和工具的位置，尤其是在空间有限、视线被遮挡时。你会用不同角度观察厨房，就像用多台相机拍摄不同视角一样，然后记住每个材料在哪里。即使有人挡住了视线，你也能凭借之前的记忆知道材料在哪个架子上，下一步该做什么。这就像论文里的多视角投影和记忆机制：通过多角度观察，结合记忆，机器人可以像你一样，灵活应对复杂的厨房场景，完成各种任务。
ELI14 Explained like you're 14
嘿，你知道吗？想象你在玩一个超级复杂的拼图游戏。你需要记住每个拼图片的样子和位置，还要根据线索猜出下一块应该放在哪里。有时候拼图被遮住了，你看不到全部的图案，这时候你就得靠之前记住的线索猜一猜。这个游戏就像机器人在做任务一样，它需要记住之前看到的东西，还要根据这些记忆来决定下一步怎么做。论文里的技术就是让机器人像你一样，记住之前的观察，利用多角度的图片和记忆，帮它在复杂的环境中找到目标，完成任务。是不是很酷？
Glossary
视觉语言模型 (Vision-Language Model)
一种结合视觉信息和语言理解的深度学习模型，能理解图像中的语义内容，并与文本进行关联。在论文中，用于空间热图预测和语义推理。
作为预训练基础，用于空间目标定位和任务理解。
多视角正交投影 (Multi-view Orthographic Projection)
将3D场景通过多个正交视角投影成二维图像，保持几何信息的空间一致性，便于模型利用预训练的视觉模型进行空间推理。在论文中用于保持几何信息的空间一致性。
实现点云到二维空间的转换，支持空间热图生成。
空间热图 (Spatial Heatmap)
一种二维概率分布图，用于表示目标在空间中的可能位置，模型通过预测热图实现空间定位。
作为空间目标定位的输出形式。
时空记忆 (Spatio-temporal Memory)
存储观察序列中的空间几何信息和交互历史的机制，支持模型在推理时利用过去信息。
增强模型在复杂任务中的连续性和遮挡恢复能力。
反投影 (Back-projection)
将二维空间中的目标位置映射回三维空间中的点，用于恢复目标的空间坐标。
实现空间目标的精确定位。
粗到细策略 (Coarse-to-fine Strategy)
先在大范围内粗略定位目标，再在局部区域内进行细粒度调整，提高操作精度。
优化空间定位的准确性。
Gaussian概率图 (Gaussian Probability Map)
用高斯函数生成的空间分布，用于表示目标的可能位置。
训练空间热图的目标。
注意力机制 (Attention Mechanism)
一种动态调整信息权重的方法，用于融合历史观察和当前特征。
在时空记忆中实现信息的有效融合。
多模态融合 (Multimodal Fusion)
结合视觉、语言、触觉等多种感知信息，提升机器人感知和操作能力。
未来研究方向之一。
迁移学习 (Transfer Learning)
将预训练模型迁移到新任务中，减少训练数据需求，提高泛化能力。
本论文中的VLM迁移到操控任务。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端遮挡和动态环境中的几何恢复能力，尤其是在长时间观察序列中保持记忆的准确性，是未来的关键挑战。当前模型在复杂场景中的鲁棒性仍有待增强，特别是在多机器人协作和多模态信息融合方面的研究还不充分。
- 2 在实际部署中，模型的计算成本和能耗仍较高，如何优化网络结构以实现更高的实时性和能效，特别是在边缘设备上运行，是未来的重要方向。
- 3 现有模型主要在模拟环境中验证，真实世界中的传感器噪声、环境变化等因素对模型性能的影响尚未充分研究，未来需要在更复杂的实际场景中进行系统性测试。
Applications
Immediate Applications
工业自动化装配
利用BridgeVLA++实现高精度、少样本的装配任务，减少人工干预，提高生产效率，适用于复杂的装配线环境。
服务机器人交互
在家庭或商业场景中，机器人可以通过自然语言指令，利用多视角记忆完成物品抓取、交互等任务，提升用户体验。
自主探索与救援
在未知或危险环境中，机器人依靠记忆推理能力，完成路径规划和目标定位，增强自主探索能力。
Long-term Vision
多机器人协作系统
实现多个机器人在复杂环境中的协同操作，通过共享记忆和空间推理，完成更复杂的任务，如协作装配或搜救。
自主学习与适应
结合强化学习和模仿学习，使机器人在新环境中自主积累经验，逐步提升操作能力，实现真正的自主智能。
Abstract
Leveraging pre-trained vision-language models (VLMs) to construct vision-language-action (VLA) models has emerged as a promising paradigm for 3D robot manipulation. However, existing 3D VLA methods remain data-hungry, exhibit limited generalization under distribution shifts, and lack explicit memory of past observations. These limitations hinder their application to data-scarce, open-world, and memory-dependent manipulation scenarios. Our previous work, BridgeVLA, improves data efficiency and generalization by preserving the input--output alignment of a pre-trained VLM during 3D action learning: raw point clouds are projected into multi-view images, and intermediate heatmaps are predicted before generating robot actions. In this work, we develop BridgeVLA++ by equipping BridgeVLA with a unified spatio-temporal memory architecture that models persistent spatial context and temporal interaction history. The resulting memory-augmented framework can reason over observation histories while preserving BridgeVLA's data efficiency and generalization capabilities. Extensive experiments show that our framework achieves strong performance on spatial manipulation tasks while exhibiting robust generalization. BridgeVLA++ further achieves state-of-the-art performance on two challenging memory-dependent manipulation benchmarks without sacrificing the data efficiency and generalization of the original BridgeVLA. In addition, BridgeVLA++ performs effectively in bimanual manipulation settings and is validated on an additional real-world robotic platform, demonstrating its scalability across tasks, environments, and robotic platforms. These results establish BridgeVLA++ as a unified 3D vision-language-action framework that simultaneously supports data-efficient learning, robust generalization, and effective memory-aware robot manipulation. Project website: https://bridgevla-plus.github.io/.
References (20)
RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo et al.
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model
Delin Qu, Haoming Song, Qizhi Chen et al.
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Wilbert Pumacay, Ishika Singh, Jiafei Duan et al.
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
Tianxing Chen, Zanxin Chen, Baijun Chen et al.
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal, Valts Blukis, Jie Xu et al.
3D-VLA: A 3D Vision-Language-Action Generative World Model
Haoyu Zhen, Xiaowen Qiu, Peihao Chen et al.
SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Haoquan Fang, Markus Grotz, Wilbert Pumacay et al.
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
Mohit Shridhar, Lucas Manuelli, D. Fox
RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
Tianxing Chen, Yuran Wang, Mingleyang Li et al.
Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-Guided 3D Policy
Ricardo Garcia, Shizhe Chen, Cordelia Schmid
SAPIEN: A SimulAted Part-Based Interactive ENvironment
Fanbo Xiang, Yuzhe Qin, Kaichun Mo et al.
Lift3D Policy: Lifting 2D Foundation Models for Robust 3D Robotic Manipulation
Yueru Jia, Jiaming Liu, Sixiang Chen et al.
Vision-Language Foundation Models as Effective Robot Imitators
Xinghang Li, Minghuan Liu, Hanbo Zhang et al.
Perceiver IO: A General Architecture for Structured Inputs & Outputs
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac et al.
X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model
Jinliang Zheng, Jianxiong Li, Zhihao Wang et al.
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Tony Zhao, Vikash Kumar, S. Levine et al.
OpenVLA: An Open-Source Vision-Language-Action Model
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al.
Fast-WAM: Do World Action Models Need Test-time Future Imagination?
Tianyuan Yuan, Zibin Dong, Yicheng Liu et al.
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
Tsung-Wei Ke, Nikolaos Gkanatsios, Katerina Fragkiadaki
BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models
Peiyan Li, Yixiang Chen, Hongtao Wu et al.