SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them
SpatialCLI employs a three-stage framework to internalize spatial perception, boosting VLM reasoning from 29.3% to 84.6% accuracy using tools, surpassing GPT-5.6 Sol.
Key Findings
Methodology
SpatialCLI adopts a Call-Learn-Internalize framework: first, it augments VLM perception by invoking spatial tools like SAM, Grounding DINO, and Depth Anything; second, it employs Cold-Start SFT and policy RL to optimize tool-use strategies; third, it converts successful tool trajectories into natural language reasoning chains via trajectory verbalization, and trains the model with dual-view internalization to embed perceptual capabilities. This integrated approach combines external perception with internal reasoning, significantly improving multi-space task performance. Experiments on Qwen3-VL-8B-Instruct show performance jumps from 29.3% to 84.6%, outperforming baseline models.
Key Results
- On SpatialCLI-Bench, the model with tools achieves 84.6% accuracy, compared to 29.3% without tools, and retains 73.8% accuracy without tools after internalization, demonstrating strong capability internalization. Performance gains exceed 50% across localization, segmentation, depth, and pose tasks.
- In complex scenarios like MindCube, tool-assisted models outperform baselines by over 20%, especially in depth and pose tasks. Post-training, models maintain high reasoning ability without external tools, confirming effective internalization.
- Compared to methods like SpaceTools and AlloSpatial, SpatialCLI shows superior multi-task coordination and perceptual internalization, validating its effectiveness in complex spatial reasoning.
Significance
This work addresses a long-standing challenge in embodied AI—integrating local spatial perception into high-level reasoning. By transforming external expert perception models into internal capabilities, it advances foundational models towards autonomous, complex spatial understanding. This progress benefits robotics, autonomous vehicles, and AR/VR applications, enabling systems to reason about their environment more effectively without relying on external tools. The internalization mechanism reduces dependency, enhances robustness, and paves the way for more intelligent, self-sufficient embodied agents capable of complex spatial tasks in real-world settings.
Technical Contribution
The paper introduces a novel three-stage framework combining external tool invocation, policy optimization, and trajectory internalization. Key innovations include: 1) multi-turn trajectory verbalization converting tool use into natural language reasoning chains; 2) dual-view training to preserve tool-use policy while internalizing perceptual capabilities; 3) comprehensive integration of spatial tools with large-scale VLMs, enabling end-to-end learning of spatial reasoning. This approach differs from prior work by systematically internalizing perceptual skills, rather than relying solely on external tools or isolated training, thus offering a new paradigm for spatial understanding in foundation models.
Novelty
This is the first systematic framework that combines external spatial tools with trajectory-based internalization, enabling models to learn perceptual capabilities directly. Unlike previous methods that treat tools as auxiliary modules, SpatialCLI internalizes these capabilities into the model’s reasoning process through trajectory verbalization and dual-view training. This innovation bridges the gap between external perception and internal reasoning, setting a new standard for multi-task spatial intelligence in foundation models.
Limitations
- The approach depends heavily on high-quality successful trajectories, which are costly to generate and may limit scalability. In highly cluttered or ambiguous scenes, perception errors can still occur, affecting internalization quality.
- The reliance on external tools introduces potential bottlenecks; if tools are unavailable or unreliable, performance may degrade. Generalization to unseen environments with different tools remains challenging.
- Training complexity and computational costs are high, requiring extensive data collection and hyperparameter tuning. Deployment in resource-constrained settings may be limited until further optimization.
Future Work
Future directions include developing more efficient trajectory sampling and internalization algorithms, reducing training costs, and enhancing generalization to unseen environments. Integrating self-supervised learning and reinforcement learning could further reduce reliance on annotated trajectories. Expanding the diversity of spatial tools and tasks will enable broader applicability, ultimately leading to embodied agents with robust, autonomous spatial reasoning capabilities suitable for real-world deployment.
AI Executive Summary
In recent years, embodied AI has made significant strides in integrating perception, reasoning, and action within complex environments. However, a persistent challenge remains: how to endow models with detailed spatial understanding that can be internally reasoned without constant reliance on external perception modules. Traditional vision-language models (VLMs) excel at semantic understanding but falter in precise spatial reasoning, especially when fine-grained localization, segmentation, depth, or pose estimation are required. Conversely, specialized perception models like SAM, VGGT, and Depth Anything provide accurate local evidence but lack the capacity for task-level reasoning and coordination.
This paper introduces SpatialCLI, a pioneering framework designed to bridge this gap by internalizing the perceptual capabilities of external spatial tools. The core idea is to leverage a three-stage process: first, during inference, the model calls external tools to gather detailed spatial evidence; second, it employs agentic fine-tuning with Cold-Start SFT and reinforcement learning to optimize tool-use strategies; third, it verbalizes successful tool trajectories into natural language reasoning chains and trains the model to internalize these capabilities via dual-view learning. This approach enables the model to perform complex spatial reasoning both with and without external tools.
Experimental results demonstrate remarkable improvements. On the SpatialCLI-Bench, performance jumps from 29.3% to 84.6% accuracy with tools, while maintaining 73.8% without tools after internalization. Similar gains are observed across diverse benchmarks like MindCube, showcasing the method’s robustness. The internalized model not only benefits from external tools but also develops an internal perceptual understanding, making it more autonomous and adaptable.
This work significantly advances the field of embodied AI by providing a scalable, systematic way to internalize spatial perception, fostering models capable of complex, multi-step reasoning in real-world scenarios. Future work will focus on expanding tool diversity, reducing training costs, and deploying these models in practical robotics and autonomous systems, pushing AI closer to human-like spatial intelligence.
Deep Dive
Glossary
Spatial Tool (空间工具)
外部模型或算法，用于提供局部空间感知信息，如定位、分割、深度估计等。
在论文中，空间工具作为外部感知模块被调用以增强VLM的空间理解能力。
Trajectory Verbalization (轨迹语言化)
将模型在空间任务中的成功工具调用轨迹转化为自然语言推理链，便于模型学习和内化。
用于将外部工具的使用过程转化为模型内部的推理能力。
Dual-View Internalization (双视角内化)
同时训练模型在有工具和无工具状态下的推理能力，确保感知能力的内化与工具策略的保持。
关键技术，用于实现感知能力的自主内化。
Cold-Start SFT (冷启动微调)
在模型初期，通过模仿专家轨迹进行微调，建立基础的工具调用策略。
为后续强化学习提供良好的起点。
Capability Internalization (能力内化)
将外部感知模型的能力转化为模型自身的内在能力，实现自主推理。
核心目标之一。
Open Questions Unanswered questions from this research
- 1 如何进一步减少对高质量轨迹的依赖，提升模型在极端复杂环境中的空间推理能力。
- 2 如何扩展空间工具的多样性，支持更复杂的空间关系和动态场景。
- 3 模型在未见环境中的泛化能力仍有限，未来需研究更强的迁移学习策略。
Applications
Immediate Applications
机器人自主导航
赋予机器人自主理解复杂空间环境的能力，无需外部感知设备，提升自主性和效率。
自动驾驶辅助系统
增强车辆对环境的空间感知，改善路径规划和障碍物避让能力。
Long-term Vision
智能空间理解平台
构建具备自主空间推理的AI平台，应用于智能制造、建筑设计等行业，推动行业智能化升级。
Abstract
Vision-language models (VLMs) are increasingly used in embodied agents to interpret visual inputs, reason about spatial relationships, and make task-level decisions based on that reasoning. However, a fundamental capability mismatch remains: general VLMs can reason about the overall task but often miss the visual details that determine success, while specialist vision models can capture those details but cannot translate them into task-level decisions. In this work, we propose SpatialCLI, a framework that teaches VLMs to reason with spatial tools and progressively internalize the specialist perceptual capabilities they provide. SpatialCLI proceeds in three stages: (1) Call exposes specialist vision models as spatial tools to augment the VLM's perception; (2) Learn uses Cold-Start SFT and agentic RL to improve tool use; and (3) Internalize verbalizes successful tool-use trajectories to internalize specialist perceptual capabilities. We further introduce SpatialCLI-Bench, a 516-example benchmark for compositional perception across localization, segmentation, depth, and pose. On MindCube, SpatialCLI raises Qwen3-VL-8B-Instruct from 29.3% to 84.6% with tools, surpassing GPT-5.6 Sol with tools (72.1%), while retaining 73.8% without tools after internalization.