Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task
Introduced O3-D dataset using depth ordering tasks, revealing models' shallow utilization of depth cues with 47-56% accuracy, highlighting significant visual-linguistic biases.
Key Findings
Methodology
This study combines depth ordering and odd-one-out psychophysical tasks to evaluate VLMs' spatial understanding. The O3-D dataset includes 37K synthetic and real images with controlled pictorial cues (height, occlusion, size, etc.) and 147K question-answer pairs. By manipulating scene parameters, the authors isolate the contribution of individual cues. Multiple prompt variations assess linguistic influence. Twelve models, including open-source and commercial, are evaluated using accuracy metrics and a novel vision-vs-language sensitivity index. The experiments also test the effects of chain-of-thought (CoT) and in-context learning (ICL).
Key Results
- Models' depth ordering accuracy ranges from 47% to 56%, close to chance, indicating poor utilization of pictorial cues. The models show minimal sensitivity to individual cues like height-in-plane and size, with performance only marginally above random. The introduced metric reveals strong linguistic biases, with answers heavily influenced by language ambiguity. ICL and CoT prompts do not significantly improve performance, suggesting static images alone are insufficient for deep spatial understanding.
- Commercial models such as GPT4-mini and Gemini 2.5 perform slightly better but still remain near chance. Combining cues yields limited gains, with the presence of height-in-plane or size cues providing the most benefit. The baseline depth map approach (DepthAnythingV2) achieves over 90% accuracy, highlighting the gap between current VLMs and specialized depth models.
- The new sensitivity index effectively quantifies the relative influence of visual cues versus language, exposing the models' over-reliance on linguistic priors. These findings underscore the need for better integration of visual spatial information and more controlled datasets to improve model robustness in 3D understanding tasks.
Significance
This research pioneers a systematic, quantitative analysis of how vision-language models perceive depth, revealing their limited cue utilization and strong language biases. The O3-D dataset and metrics provide standardized benchmarks for future development, addressing a critical gap in multimodal spatial reasoning. The insights gained can guide the design of more perceptually grounded models, impacting applications like robotics, autonomous navigation, and AR/VR. By exposing current shortcomings, this work pushes the community toward models capable of genuine 3D understanding, essential for real-world scene comprehension.
Technical Contribution
The paper introduces a psychophysics-inspired evaluation framework combining synthetic scene control with real-world data, enabling precise cue-level analysis. It develops a novel metric to quantify visual versus linguistic sensitivities, facilitating detailed bias analysis. The dataset encompasses nine pictorial cues and their interactions, with controlled variations in cue strength and referring expression clarity. The methodology allows systematic ablation studies, revealing the limited cue utilization and persistent language bias across diverse models. This approach bridges cognitive science and AI, offering a new paradigm for evaluating spatial understanding.
Novelty
This is the first comprehensive dataset (O3-D) to systematically control and evaluate individual pictorial depth cues in both synthetic and real images within a multimodal context. It combines psychophysical tasks with large-scale question-answer pairs, enabling fine-grained analysis of cue utilization and linguistic bias. The introduction of a vision-vs-language sensitivity metric is a novel contribution, providing a quantitative measure of model biases. Unlike prior datasets, O3-D isolates specific cues and their interactions, offering unprecedented insight into models' spatial reasoning capabilities.
Limitations
- Models evaluated are primarily static, trained on fixed datasets, and may not generalize well to dynamic or real-world scenarios involving motion or temporal cues.
- Simulated scenes, despite their control, cannot fully replicate the complexity of real environments, potentially limiting ecological validity.
- The sensitivity metric, while effective, does not reveal internal model mechanisms, necessitating further neuro-cognitive studies for deeper understanding.
Future Work
Future research will extend to dynamic scene understanding, incorporating temporal cues and multi-step reasoning. Developing models that integrate visual cues more effectively, possibly via multi-task learning or reinforcement strategies, is a priority. Expanding datasets to include more complex, cluttered, and real-world environments will improve robustness. Additionally, integrating neuro-inspired architectures could enhance models' spatial cognition, bridging the gap between AI and human perception.
AI Executive Summary
This study addresses a fundamental challenge in multimodal AI: understanding how vision-language models perceive and utilize spatial information. Despite their impressive performance in many tasks, current models show a striking deficiency in depth perception, with accuracy barely surpassing chance levels of 50%. To systematically evaluate this, the authors introduce the O3-D dataset, a large-scale collection of synthetic and real images explicitly designed to control for nine key pictorial depth cues such as height-in-plane, occlusion, and size. These cues are manipulated independently, allowing precise analysis of their influence on model responses.
The core methodology involves psychophysical tasks—depth ordering and odd-one-out—that mimic human perceptual experiments. By varying the clarity of referring expressions and cue combinations, the authors assess how models integrate visual and linguistic information. The results reveal that most models, including state-of-the-art commercial and open-source systems, perform only marginally better than random guessing, with accuracies between 47% and 56%. Furthermore, the models exhibit strong linguistic biases, often ignoring visual cues altogether.
An innovative metric introduced in this work quantifies the relative sensitivities of models to visual cues versus language priors. This metric exposes a prevalent over-reliance on linguistic cues, which hampers true spatial understanding. Additional experiments with chain-of-thought and in-context learning prompts show limited improvements, underscoring the challenge of static image-based learning.
Overall, this research provides a critical benchmark and analytical framework for the community, highlighting the gap between current AI capabilities and human-like 3D scene understanding. The datasets, evaluation tools, and insights offered will guide future efforts toward more perceptually grounded models, with broad implications for robotics, autonomous systems, and immersive technologies. Despite its limitations, such as the focus on static scenes, this work marks a significant step toward robust spatial reasoning in multimodal AI systems.
Deep Analysis
Background
近年来，深度感知在计算机视觉中逐渐成为核心研究内容。早期工作如Eigen等（2014）提出单目深度估计，结合结构光与多视角重建技术，推动了三维场景理解的发展。随后，深度排序和空间关系分析成为评估模型空间认知的主流方法（如Zhou et al., 2019）。然而，这些方法多依赖真实场景数据，缺乏对单一深度线索的控制，难以揭示模型对不同线索的利用机制。近年来，视觉-语言模型（VLM）崛起，试图在静态图像中实现空间理解（如Li et al., 2020），但其深度感知能力仍未充分验证。现有数据集（如DepthCues, 2015）虽涵盖部分线索，但缺乏系统控制与多线索交互分析。本文基于心理物理学中的深度排序和奇异物识别任务，结合多线索控制，填补了这一研究空白。
Core Problem
尽管VLM在多模态任务中表现优异，但其空间理解能力仍然有限。模型在深度排序任务中的准确率仅为47%-56%，远低于人类水平，显示其未能充分利用图像中的深度线索。此外，模型偏向语言表达，忽视视觉空间信息，导致在复杂场景中的空间推理能力不足。这一问题限制了模型在机器人导航、增强现实等实际应用中的表现。现有评估方法多关注整体性能，缺乏对线索利用层级和偏差的细粒度分析，亟需设计更具控制性和解释性的评估框架。
Innovation
本研究的创新点包括：1）结合心理物理学中的深度排序任务，设计了控制九种深度线索的合成与真实场景数据集（O3-D），实现对模型线索利用的系统分析；2）提出视觉-语言敏感性指标，量化模型对视觉线索与语言表达的偏差，揭示模型偏向语言的普遍现象；3）多样化提示设计，评估模型在不同语言模糊度下的表现，增强评估的全面性。这些创新为多模态空间理解提供了新的分析工具和数据基础，有助于推动模型在空间推理上的突破。
Methodology
- �� 构建3D场景：结合模拟与真实环境，控制目标对象在不同深度平面。• 生成2D视图：通过调整相机位置，加入或排除九种深度线索（如遮挡、大小、光影等）。• 设计提示：多样化问题模板，变换目标指称的清晰度与描述方式。• 数据集制作：合成与真实场景共计37K图像，配合147K问答对。• 模型评估：测试12个模型，采用深度排序准确率和偏差指标。• 统计分析：比较模型对不同线索的利用情况，分析视觉与语言偏差。• 实验还包括ICL和CoT提示的效果验证，评估模型在多模态推理中的能力。
Experiments
采用合成场景（Kubric）与真实场景结合，控制九种深度线索，生成多样化视图。评估模型包括深度排序准确率、视觉-语言敏感性指标和偏差分析。模型涵盖主流开源（如LLaVA、BLIP2）与商业（如GPT4-mini）模型，采用多样化提示和问答格式。通过对比不同线索组合、提示策略，验证模型在空间理解中的表现。还进行了ablation研究，分析线索交互与偏差变化，确保结果的稳健性。
Results
所有模型深度排序准确率均在47%-56%，明显低于基线DepthAnythingV2（约93%），显示模型未充分利用深度线索。模型对高度、大小等线索的敏感性不足，偏向语言表达。引入视觉-语言偏差指标，发现模型普遍偏向语言线索，ICL和CoT未带来显著提升。模型在不同线索组合下表现差异明显，单一线索利用率极低，强调空间理解的难度。结果揭示了当前VLM在空间推理上的瓶颈，为未来改进提供了定量依据。
Applications
该数据集和评估指标可应用于机器人导航、增强现实、虚拟现实等领域，帮助模型更好理解空间关系。未来可结合动态场景、多模态交互，提升模型在复杂环境中的空间认知能力。还可用于训练更具空间推理能力的多模态模型，推动智能系统在实际场景中的应用。
Limitations & Outlook
模型在静态图像上的表现受限，未考虑动态场景与时序信息，未来需结合视频理解提升空间推理能力。模拟场景虽丰富，但难以完全还原真实环境的复杂性，可能影响模型泛化。指标虽能区分视觉与语言偏差，但未能揭示模型内部认知机制，需结合神经科学研究深入分析。
Plain Language Accessible to non-experts
想象你在厨房里准备做饭。厨房里有各种工具和食材，有的在柜子里，有的在桌子上。你需要判断哪块肉离你更近，哪块在更远。以前，你会根据肉的大小、颜色和阴影猜测距离，但这些线索有时会误导你。现在，科学家让电脑像你一样用不同的线索判断距离。他们设计了各种厨房场景，控制每个线索的出现，测试电脑是否能正确判断距离。结果发现，电脑大多只听语言描述，忽略了实际的空间线索，就像你只听别人说肉在桌子上，却没注意到它的大小或阴影。这帮助我们理解电脑在空间认知上的不足，也为未来让它们更聪明提供了方向。
ELI14 Explained like you're 14
想象你在玩一个游戏，你的任务是找出哪个玩偶离你更近或更远。你可以用玩偶的大小、阴影、遮挡等线索来判断，但有时这些线索会让你迷糊。科学家们也遇到类似的问题，他们用电脑帮忙，但发现电脑很难用这些线索判断距离。于是，他们设计了一个特别的“测试场景”，让电脑在不同图片中练习用各种线索判断距离。结果显示，电脑大部分时间都只听语言描述，忽略了图片中的空间线索，就像你只听别人说玩偶在桌子上，却没注意到它的大小或阴影。这告诉我们，电脑在理解空间关系方面还很弱，需要更多的训练和改进。未来，希望让电脑像人一样聪明，能真正理解空间中的距离和结构，就像你能轻松找到哪个玩偶更靠近你一样！
Abstract
In this paper, we study depth perception of vision-language models (VLMs) to isolate the effects of pictorial depth cues and disentangle vision and language influences on model performance. To this end, we combine depth-ordering and odd-one-out psychophysical tasks: the VLMs are presented with images where one object is at different depth relative to other, otherwise identical, objects, and must determine whether the odd-one-out target is closer or farther to the observer. To create stimuli, we generate 2D views from simulated and real 3D scenes while controlling the presence of individual pictorial depth cues, enabling a fine-grained analysis of cue-level contributions. Language effects are examined by varying referring expression clarity. We also introduce a novel metric to quantify vision-vs-language sensitivities. Applying this methodology, we create the Odd-One-Out Depth (O3-D) dataset with 37K real and synthetic images and 147K image-question pairs. Evaluation of 12 open-source and commercial models on O3-D shows under-utilization of depth cues and depth-ordering accuracies between 47% and 56%, with no model above chance level. At the same time, our metric reveals strong linguistic bias in the answers. Neither chain-of-thought (CoT) nor in-context learning (ICL) significantly improves performance, suggesting that static image data alone may be insufficient for depth understanding. All code, the image generation pipeline, and the O3-D dataset are publicly released at https://github.com/lyiqian/o3-d.