Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

Recent advances in robotic perception, exemplified by large-scale multimodal models like CLIP and Florence, have improved scene understanding. Traditional robotic manipulation relies on static cameras and local spatial reasoning, which limits performance in cluttered or dynamic environments. Multi-view perception techniques, including geometric pose estimation (VGGT) and multi-camera fusion, have been explored but lack persistent scene representations. Spatial memory mechanisms, such as Neural Map and Memory-Augmented Networks, have been proposed to enhance scene persistence, yet they often focus on static or local scenes. The integration of multi-view spatial memory with deep semantic features remains an open challenge, crucial for enabling robots to operate beyond immediate visual input in complex real-world scenarios.

Current VLA models assume the target object is always visible within the current camera view, which is unrealistic in cluttered or dynamic environments. When objects are occluded or outside the field of view, models lack the ability to reason about their location, leading to task failure. This view-dependent limitation restricts the robot's operational scope, especially in multi-step or multi-object tasks requiring persistent scene understanding. Overcoming this bottleneck demands a mechanism to build and maintain a global, multi-view scene representation that supports reasoning about occluded or out-of-view objects, ensuring robust manipulation in real-world settings.

The paper introduces a multi-view spatial memory framework that fuses geometric pose estimation (VGGT) with deep semantic features (YOLO, DINOv3). This memory is constructed through systematic scene scanning, integrating multi-view observations into a unified, object-centric map. A similarity-aware fusion mechanism ensures temporal consistency during scene updates, while cross-attention-based retrieval aligns instructions with scene context. These innovations enable the robot to maintain a persistent, globally consistent understanding of the environment, supporting out-of-vision manipulation and reducing reliance on reactive, view-dependent perception. This approach significantly advances the state-of-the-art in autonomous scene reasoning.

• �� Spatial Memory Construction: Using a movable head camera, the robot scans the environment, estimates camera poses with VGGT, and extracts semantic features via YOLO and DINOv3. These features are lifted into 3D space and fused into a unified scene memory. • Dynamic Memory Refinement: During manipulation, new observations are integrated into the scene memory through similarity-weighted fusion, maintaining consistency over time. • Contextual Memory Retrieval: Cross-attention modules align instruction embeddings with scene memory, activating relevant spatial regions to guide manipulation. • Scene Representation: Object-level appearance features and 3D geometries are combined into memory tokens, forming a compact, geometry-aware scene map. • Real-time Updates: The memory is iteratively refined with incoming data, supporting robust reasoning under partial observability.

The authors evaluate SOMA on five real-world out-of-vision pick-and-place tasks with increasing difficulty, involving occlusion, sequential steps, and dual-arm coordination. Success rates, localization times, and viewpoint search efforts are measured, showing significant improvements over baseline models. Additional tests on RoboCasa GR1 and SimplerEnv validate the generality of the memory mechanism under fully observable conditions. Ablation studies compare models with and without multi-view scanning and memory refinement, confirming the contribution of each component. Hyperparameters, such as scan frequency and fusion thresholds, are tuned for optimal performance, ensuring robustness across scenarios.

SOMA achieves a success rate of 28.3%, outperforming models without persistent memory by nearly 10%. It reduces target localization time by 55%, viewpoint search by 50%, and enables near one-shot grasping even with partial occlusion. In multi-step and dual-arm tasks, the model maintains scene consistency and reduces error accumulation. Ablation results show that multi-view scanning alone improves success by 4%, while full memory refinement adds another 4.2%, demonstrating the effectiveness of the integrated approach. The system's robustness is further validated in dynamic, cluttered environments, confirming its practical viability.

This framework is directly applicable to autonomous warehouse robots, service robots in homes, and manufacturing automation, where targets often move out of view. It enables robots to perform complex, multi-step tasks with minimal re-scanning, reducing operational latency. The ability to maintain persistent scene understanding also benefits long-term deployment in unstructured environments, facilitating tasks like object retrieval, sorting, and assembly. Future integration with reinforcement learning could further enhance adaptability and efficiency, making robots more autonomous and reliable in real-world applications.

The multi-view scanning process introduces latency, limiting real-time responsiveness. Scene dynamics faster than memory updates can cause inaccuracies, especially in highly cluttered or rapidly changing environments. The approach relies on accurate geometric estimation; errors in pose estimation can propagate, affecting scene consistency. Scalability to larger environments remains a challenge, requiring more efficient memory management and faster perception algorithms. Future work should address these issues to improve robustness and applicability in diverse real-world scenarios.