SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models
SpatioLM enhances vision-language models' spatial reasoning using a plug-and-play module, achieving 71.6 on VSI-Bench without extra 3D inputs.
Key Findings
Methodology
SpatioLM employs a frozen pretrained VLM backbone, inserting a lightweight Spatio-Vision module composed of stacked blocks. It extracts intermediate vision tokens (from layer 16 of ViT) and leverages pseudo depth and camera supervision to train dense depth and ray map predictions. The module uses an alternating attention mechanism to model intra-frame details and inter-frame geometric consistency, enabling the model to elicit implicit 3D spatial knowledge without altering the core model. During inference, only the module is activated, maintaining efficiency. The training combines language modeling loss, token distillation, and geometric supervision, resulting in a model capable of high-quality spatial perception and understanding.
Key Results
- On VSI-Bench, SpatioLM scored 71.6, surpassing all previous models and setting a new state-of-the-art. It achieved 83.5% on MD-S and 69.0% on MD-M depth benchmarks, outperforming existing methods by significant margins. The model also excelled in downstream tasks like embodied manipulation, demonstrating strong transferability. Ablation studies confirmed the importance of intermediate tokens and alternating attention. The model's ability to generate physically coherent depth and spatial relations was validated through extensive evaluations.
- Compared to baseline models, SpatioLM consistently improved performance across perception and understanding tasks, with notable gains in depth accuracy, spatial relation reasoning, and route planning success rates. Its robustness was verified across multiple datasets, including synthetic and real-world scenes, indicating broad applicability.
- The integration of pseudo supervision signals and the non-invasive design allowed the model to learn meaningful 3D representations while preserving the general semantic capabilities of the pretrained VLM, avoiding catastrophic forgetting. This approach offers a scalable, efficient pathway for enhancing spatial reasoning in large multimodal models.
Significance
This work addresses a fundamental challenge in multimodal AI: how to endow vision-language models with robust spatial reasoning without relying on costly external 3D sensors or invasive fine-tuning. By leveraging the inherent structure of pretrained models and introducing a parameter-efficient, plug-and-play module, it opens new avenues for deploying spatially aware AI in real-world applications such as robotics, autonomous navigation, and AR. The approach balances performance, scalability, and generality, representing a significant step toward truly spatially intelligent multimodal systems. It reduces hardware dependencies and simplifies integration, making advanced spatial reasoning accessible for broader AI deployment.
Technical Contribution
The paper introduces a novel framework that exploits the intermediate layer tokens of frozen VLMs, combined with a lightweight, alternating-attention Spatio-Vision module. This module elicits geometric features by modeling intra-frame details and cross-frame consistency, guided by pseudo depth and camera supervision. The key technical innovation is the zero-initialized projection fusion, which integrates geometry-aware tokens into the frozen language model without disrupting its pretraining distribution. The architecture achieves high efficiency by only training the spatial module, preserving the core model's generality. The dual dense prediction head further enforces explicit geometric learning, enabling accurate depth and ray map estimation. This design significantly advances the state-of-the-art in parameter-efficient spatial reasoning.
Novelty
This is the first work to utilize intermediate vision tokens from a frozen VLM to elicit 3D geometric knowledge without external 3D inputs or spatial encoders. Unlike prior approaches relying on explicit geometric priors or external encoders, it leverages the model’s inherent structure through a plug-and-play module, avoiding catastrophic forgetting. The alternating attention mechanism within the Spatio-Vision blocks captures both intra-frame details and inter-frame consistency, offering a new paradigm for spatial reasoning in large-scale multimodal models. This approach balances performance, efficiency, and generality, marking a significant innovation in the field.
Limitations
- The reliance on pseudo depth and camera supervision means that errors in these signals can affect spatial reasoning accuracy, especially in highly dynamic or occluded scenes.
- While parameter-efficient, the added module still incurs computational overhead, which may challenge real-time deployment in resource-constrained environments.
- The current design focuses on static scenes; extending to dynamic, multi-view, or highly cluttered environments remains an open challenge, requiring further research.
Future Work
Future directions include integrating multi-view consistency, enhancing robustness in dynamic scenes, and exploring self-supervised learning for geometric features. Extending the framework to real-time applications and multi-task learning for navigation, manipulation, and scene understanding will further broaden its impact. Additionally, combining this approach with reinforcement learning could enable autonomous agents with stronger spatial reasoning capabilities, pushing the boundary of embodied AI.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个厨房里准备做饭。厨房里有很多不同的工具和食材，但你不需要专门的地图或传感器来知道它们的位置。你只用眼睛观察，就能知道锅在哪儿，刀在哪儿，菜在桌子上。这个过程就像论文里的SpatioLM，它不用额外的传感器，只靠观察图片中的物体，自己学会它们之间的距离和关系。它还会用一些虚拟的“深度”信息，帮助自己更清楚地知道哪个东西离自己更近。这样一来，它就能更好地导航、操作，甚至帮你做菜。这个方法让机器变得更聪明，更像人一样用眼睛看世界，理解空间关系。未来，这样的技术可以让机器人在复杂的环境中工作得更好，比如自动驾驶汽车或智能家居，变得更加自主和智能。
ELI14 Explained like you're 14
想象你在玩一款超级酷的游戏，你的角色需要在房间里找到隐藏的宝藏。以前，你只能靠猜测或者用特殊的地图来知道宝藏在哪，但这很麻烦，也不总是准。现在，有一种新方法，就像你有一双会“看见”空间的眼睛，不用额外的地图，只靠观察房间里的东西，就能知道它们之间的距离和位置。这就像论文里的SpatioLM，它用一种聪明的方式，从图片里提取信息，然后自己学会理解空间关系。它还会用一些虚拟的“深度”信息，帮自己更好地判断距离。这样一来，它就能帮你更快找到宝藏，甚至在复杂的房间里也能表现得很棒。这就像让机器变得更聪明，能自己理解空间，帮人类完成各种任务，未来会变得更酷！
Abstract
Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient \textit{\textbf{Spatio}-vision \textbf{L}anguage \textbf{M}odels (SpatioLM)}, that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks. Code is available at \href{https://github.com/xiaomi-research/spatio-lm}{\faGithub~spatio-lm}.