LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
LAST integrates visual tools to enhance spatial-temporal reasoning, boosting VLMs' performance on 3D and long video understanding by 15.8% and 8.3% respectively.
Key Findings
Methodology
LAST employs external visual tools—such as object tracking, spatial grounding, and depth estimation—to construct continuous visual chains representing space and time. These chains are integrated into the reasoning process of general VLMs, either via zero-shot prompting (e.g., GPT-4o) or fine-tuning (e.g., LAST-7B). The approach involves generating visual thinking trajectories, which are then used to improve the model’s understanding of 3D structures and long videos. The framework combines visual tool outputs with language reasoning, forming a multi-modal chain of thought that addresses the limitations of text-only chain-of-thought methods. Extensive datasets with 48K text-based and 124K visual trajectories support training, leading to significant performance gains across multiple benchmarks such as EgoSchema, VSI-Bench, and NExT-QA.
Key Results
- In zero-shot settings, LAST boosts GPT-4o’s accuracy on EgoSchema from 69.6% to 85.4%, a 15.8% increase, and improves VSI-Bench results by 8.3%.
- Fine-tuning LAST-7B yields 41.3% accuracy on VSI-Bench, surpassing baseline Qwen2.5-VL-7B’s 33.0%. On NExT-QA, it reaches 86.2%, outperforming previous models.
- Across spatial and video understanding tasks, LAST consistently outperforms existing methods, validating its generality and effectiveness in multi-modal reasoning.
Significance
This work addresses the core challenge of enabling general VLMs to reason about 3D space and extended videos without specialized architectures or inputs. It bridges the gap between visual perception and language reasoning, bringing AI closer to human-like understanding. The approach significantly advances the field of multimodal AI, with broad implications for autonomous systems, robotics, and multimedia analysis. By integrating visual chains into reasoning, models can better interpret complex environments, leading to more robust and versatile AI systems. The methodology also opens new avenues for research in visual cognition and multi-modal learning, emphasizing the importance of visual thinking trajectories in AI development.
Technical Contribution
The paper introduces a novel framework that combines external visual tools with language models to construct visual chains of thought, effectively capturing spatial and temporal information. It develops a scalable data collection pipeline for visual reasoning trajectories and demonstrates how to incorporate these into pre-trained models via fine-tuning. The key technical innovation lies in the dynamic generation and integration of visual chains during inference, enabling models to perform complex reasoning tasks without architectural modifications. The approach also provides theoretical insights into the importance of visual reasoning pathways, validated through extensive benchmark evaluations showing substantial performance improvements.
Novelty
This is the first systematic effort to embed visual reasoning chains into generalist VLMs, moving beyond traditional text-only chain-of-thought paradigms. Unlike prior work that relies solely on textual prompts, LAST actively constructs visual trajectories using external tools, enabling models to think in space and time. This represents a fundamental shift in multi-modal reasoning, leveraging external visual modules to simulate human-like spatial-temporal cognition within a unified framework. The method’s ability to improve diverse benchmarks without task-specific architecture marks its novelty and broad applicability.
Limitations
- 依赖外部视觉工具的准确性，工具误差可能引入噪声，影响推理效果，尤其在复杂场景中表现不佳。
- 视觉链的生成和调用增加了推理时间和计算成本，不适合实时应用场景。
- 在极端复杂或超长视频场景中，模型仍面临理解瓶颈，未来需优化视觉链的鲁棒性和效率。
Future Work
未来将探索端到端训练视觉链生成机制，减少对外部工具的依赖，实现更自主的空间-时间推理。结合多模态数据增强模型理解能力，扩展到三维重建和动态场景理解。还计划引入自监督学习策略，降低数据标注成本，推动模型在实际环境中的应用落地。
AI Executive Summary
Understanding the spatial and temporal structure of visual scenes is fundamental to human cognition, yet current vision-language models (VLMs) struggle with these aspects, especially in 3D environments and long videos. Traditional approaches often rely on task-specific architectures or additional inputs like depth maps or 3D data, limiting their generality and scalability. This paper introduces LAST (LeArning to Think in Space and Time), a novel framework that enhances general VLMs by integrating external visual tools—such as object trackers, spatial grounders, and depth estimators—into the reasoning process. These tools generate visual chains that represent continuous space-time trajectories, which are then fused with language reasoning to form a multi-modal chain of thought. The approach is validated through extensive experiments on benchmarks like EgoSchema, VSI-Bench, and NExT-QA, demonstrating significant improvements in accuracy—up to 15.8% in zero-shot settings and 8.3% after fine-tuning. Notably, LAST achieves comparable or superior performance to specialized 3D models without requiring additional inputs or architectural modifications, highlighting its versatility and scalability. The results underscore the importance of visual thinking trajectories, opening new avenues for multi-modal AI that more closely mimics human spatial-temporal reasoning. Future work will focus on end-to-end visual chain generation, reducing reliance on external tools, and expanding to more complex dynamic environments, aiming to push the boundaries of AI understanding in real-world scenarios.
Deep Dive
Abstract
Humans can perceive and understand 3D space and long videos from sequential visual observations. But do vision-language models (VLMs) can? Recent work demonstrates that even state-of-the-art VLMs still struggle to understand 3D space and long videos, although they are powerful in typical vision-language tasks. Current methods often rely on specialized architectural designs to improve performance for 3D tasks and video understanding tasks separately. In contrast, we propose LAST, short for LeArn to Think in Space and Time, to jointly improve 3D spatial and long video understanding for general VLMs with only a set of 2D images as inputs. LAST makes VLMs think in space and time rather than only with text before giving the final answer, building visual thinking trajectories in 3D space and temporal dimension. We demonstrate the effectiveness of LAST in two scenarios: 1) zero-shot, where we directly prompt proprietary models; and 2) fine-tuning general VLMs with data that include thinking trajectories in 3D space and time. We show that LAST brings substantial gains in various benchmarks, including 3 spatial understanding, 4 video understanding, and 3 image understanding tasks. Notably, 15.8% gains on EgoSchema with GPT-4o in a zero-shot manner and 8.3 gains on VSI-Bench compared with Qwen2.5-VL-7B.