Do vision-language models search like humans? Reasoning tokens as a reaction-time analog in classic visual-search paradigms
Using reasoning tokens as a proxy for reaction time, the study analyzes vision-language models' search behaviors across classic paradigms, revealing partial human-like signatures.
Key Findings
Methodology
The study adapts classic visual search paradigms—feature vs. conjunction, spatial configuration, enumeration, and asymmetry—testing models like GPT-4, GPT-5.5, and Claude. Since models lack reaction time, the number of reasoning tokens used per trial serves as an effort proxy. Experiments involve multiple API calls with images and prompts, recording output tokens. Effort curves are compared with human reaction time data (Wolfe et al., 2010). Control experiments with blurring validate effort reflects search rather than recognition difficulty. The approach enables behavior-based evaluation without internal access, capturing serial or parallel search signatures.
Key Results
- Frontier models (GPT-5.5) exhibit flat effort in feature search and increasing effort with set size in conjunction search, matching human reaction-time slopes. Accuracy remains near-ceiling for advanced models, while mid-tier models collapse to chance in large sets, confirming capability differences. In search asymmetry tasks, GPT-5.5 shows higher effort in the hard (target-absent) direction, reversing human patterns; Claude shows high error rates instead. Enumeration tasks reveal models maintain high accuracy but increase tokens, indicating reliance on computation rather than error-prone counting.
- Effort effort curves align with human reaction times, especially in the strongest models, demonstrating serial search signatures. The effort in conjunction tasks grows monotonically with set size, with GPT-5.5 reaching 4.98 tokens/item. The reversal in effort between target-present and -absent conditions contrasts human data, indicating models confirm targets differently. In spatial configuration tasks, effort increases with set size for GPT-5.5, but remains minimal for Claude, reflecting different strategies. These results highlight both similarities and divergences in search behaviors.
- The models’ behaviors suggest that advanced VLMs partially emulate human visual search signatures, especially in effort patterns, but differ in strategies for target absence and asymmetry. These differences reveal underlying cognitive mechanisms and limitations, informing future model development. The approach offers a behavioral benchmark for AI cognition, bridging psychophysics and machine learning, with implications for designing more human-like perception systems.
Significance
This work pioneers the behavioral evaluation of vision-language models using psychophysical paradigms, revealing their partial mimicry of human search signatures through effort proxies. It demonstrates that models organize internal computation along difficulty axes similar to human attention, providing a new framework for understanding AI cognition. The findings bridge cognitive science and AI, offering insights into how models process complex visual information. By identifying divergences, the study guides future improvements toward more human-like perception, crucial for applications like autonomous systems, robotics, and assistive AI. The methodology also opens pathways for behavior-based AI evaluation, complementing traditional accuracy metrics.
Technical Contribution
The study introduces the innovative use of reasoning tokens as an effort measure, enabling behavior-based analysis without internal model access. It systematically applies classic visual search paradigms—feature vs. conjunction, spatial configuration, enumeration, and asymmetry—to VLMs, revealing their search strategies. The approach quantifies effort curves and correlates them with human reaction times, establishing behavioral signatures of serial and parallel search. It also incorporates control experiments (blurring) to validate effort as search effort. This framework extends psychophysical paradigms into AI evaluation, providing a new quantitative tool for understanding model cognition and strategy, with potential to influence model design and interpretability.
Novelty
This is the first work to adapt classic psychophysical visual search paradigms to evaluate vision-language models via effort proxies (reasoning tokens). It innovatively bridges cognitive psychology and AI, moving beyond accuracy metrics to behavior-based signatures. The reversal of effort patterns in target presence/absence and the systematic analysis across multiple paradigms constitute a novel contribution, offering a new lens to interpret model cognition. The methodology provides a scalable, non-intrusive way to probe internal search strategies, setting a foundation for future behaviorally grounded AI assessment.
Limitations
- Using reasoning tokens as effort proxies, while insightful, cannot fully replicate human reaction times due to differences in internal mechanisms and training biases. The effort measure may reflect model-specific strategies rather than pure search effort.
- The experiments focus on a limited set of models and tasks; broader validation across diverse architectures and real-world scenarios is needed to generalize findings.
- Control experiments with blurring address perceptual difficulty but do not encompass complex visual environments with dynamic or cluttered scenes, limiting ecological validity. Future work should incorporate richer, more realistic visual contexts.
Future Work
Future research will extend behavioral paradigms to dynamic and multimodal tasks, exploring how models adapt to real-world visual complexities. Incorporating more diverse architectures, such as vision transformers with different attention mechanisms, will test the universality of observed signatures. Developing more refined effort metrics aligned with neural or cognitive models could deepen understanding of internal search strategies. Additionally, integrating these behavioral insights into model training may foster more human-like perception, advancing AI's applicability in autonomous systems, robotics, and assistive technologies.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在找厨房里的某个调料瓶。有时候，你一眼就能找到，比如盐瓶，因为它颜色特别；这就像特征搜索。可是，有时候两个瓶子长得很像，比如两个调料瓶只差一个标签，你得一个个仔细看，像结合搜索。还可能你要数一数盘子里有几个苹果，像枚举任务。更复杂的是，要找一根倾斜的胡萝卜，和找一根直的，难度不同。科学家用电脑模拟这个找东西的过程，看看它们是不是像人一样，花费的“努力”是否随着任务变难而增加。这个研究帮助我们理解，机器是不是在用和人类一样的“脑袋”在找东西。
ELI14 Explained like you're 14
想象你在玩“找东西”的游戏，你要在一堆玩具里找到特定的那个。有时候，这个玩具有特别的颜色或形状，很快就能找到，就像人用快速的“特征搜索”；但有时候，两个玩具长得很像，你得仔细看每个细节，像“结合搜索”。如果你还要数一数有多少个特定的玩具，就像在用眼睛数苹果一样。科学家用电脑里的模型模拟这个“找目标”的过程，看看它们是不是像人一样，花费的时间和努力会随着目标变多变复杂而变化。这个研究帮我们理解，机器是不是在用和人类一样的“脑袋”在找东西。
Abstract
Visual search has been one of the most productive paradigms in the study of visual attention: the way reaction time scales with the number of items distinguishes parallel, "pop-out" search from serial, attention-demanding search. I ask whether vision-language models (VLMs) exhibit the same behavioral signatures. I adapt four classic paradigms: feature versus conjunction search, spatial-configuration (T-vs-L) search, enumeration, and the tilted/vertical search asymmetry; and present them to current frontier and mid-tier models. Because a single model call has no reaction time, I use the number of reasoning ("thinking") tokens a model spends per trial as a within-model analog of search effort, and I compare against a large public human benchmark (Wolfe et al., 2010). The models reproduce several human signatures: feature search costs flat effort while conjunction effort climbs with set size; frontier models hold accuracy where mid-tier models collapse to chance; and a resolution control shows the conjunction cost is genuine search rather than difficulty resolving small shapes. They also diverge from humans in informative ways. The target-present effort slope exceeds the target-absent slope, reversing the human ordering; enumeration remains accurate where humans would lose count; and a reasoning model with adaptive deliberation declines to deliberate on detection tasks altogether, so that a single search expresses itself as an effort gradient in one model and as an accuracy cliff in another. I argue that psychophysical paradigms, applied behaviorally, are a sharp and inexpensive probe of machine visual cognition, and that the points of divergence are as informative as the points of agreement.