Describing Images $\textit{Fast and Slow}$: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes
Using Dutch image descriptions and eye-tracking data, this study quantifies human visuo-linguistic signal variation and evaluates pretrained models' ability to capture it.
Key Findings
Methodology
The study employs the DIDEC corpus, integrating eye-tracking and speech data to quantify variation in speech onset, description diversity, and gaze patterns. Spearman correlation analyses reveal significant links between visual and linguistic signals. Visual features from pretrained models (CLIP, ViT) are used in a similarity prediction framework to assess their capacity to encode human-like variability. Results indicate that these models weakly to moderately capture such variation, highlighting their limited bias understanding of cognitive complexity.
Key Results
- Average speech onset across images was 3.42 seconds with a maximum of 25.37 seconds, indicating substantial individual differences. The correlation between description similarity (BLEU-2) and speech onset was moderate (ρ=-0.516), showing that earlier descriptions tend to be more similar. Gaze variation correlated positively with speech delay (ρ=0.455), suggesting that gaze variability accompanies longer response times. Visual features from pretrained encoders showed limited correlation (~0.4), underscoring their weak grasp of human cognitive biases.
- These findings emphasize the influence of image complexity on behavioral variability. The models' limited sensitivity to such signals suggests a need for integrating human behavioral cues into training to improve naturalness and diversity in generated descriptions.
Significance
This work pioneers the systematic quantification of human visuo-linguistic variability, linking visual complexity to behavioral signals. It underscores the importance of modeling human cognitive biases for more natural multimodal AI systems. By revealing the limited capacity of current pretrained encoders to capture these signals, it advocates for incorporating human behavioral data into model training, which could significantly enhance the authenticity and diversity of AI-generated language, impacting fields like assistive tech, human-computer interaction, and cognitive modeling.
Technical Contribution
The paper introduces a novel multimodal variability quantification framework combining eye-tracking and speech data. It innovatively applies similarity prediction techniques with pretrained visual encoders (CLIP, ViT) to assess their capacity to encode human-like behavioral variation. This approach provides a new lens for analyzing model biases and their alignment with human cognition, offering a methodological advancement over traditional feature-based or regression-based analyses. It also establishes a foundation for future research on integrating behavioral signals into multimodal AI systems.
Novelty
This is the first comprehensive study to combine eye movement and speech onset data to quantify visuo-linguistic variation. It innovatively employs pretrained visual encoders in a similarity prediction framework to evaluate their ability to encode human behavioral variability. Unlike prior work focusing solely on linguistic or visual features, this research emphasizes the importance of behavioral signals, filling a crucial gap in understanding how AI models can better mirror human cognition and variability.
Limitations
- The models' limited ability to capture complex visual-cognitive biases suggests that current pretrained encoders lack sufficient training on behavioral signals. The study is confined to Dutch data, limiting cross-linguistic generalization. Eye-tracking and speech signals are only partially synchronized, and additional cognitive metrics could improve analysis. The experimental setup does not fully explore complex or ambiguous scenes, which are critical for real-world applications.
Future Work
Future research should incorporate richer cognitive measures, such as reaction times and neural signals, to better model human variability. Developing adaptive training strategies that embed behavioral cues into multimodal models could enhance their naturalness. Cross-linguistic studies and more complex scene analyses are needed to generalize findings. Additionally, integrating reinforcement learning could enable models to learn from human feedback, improving their sensitivity to cognitive biases and variability.
AI Executive Summary
This research leverages the Dutch Image Description and Eye-tracking Corpus (DIDEC) to explore the intricate relationship between visual complexity and human behavioral signals during image description. By analyzing eye movements and speech onset times across 307 real-world images, the study quantifies the variability in how humans initiate and produce descriptions. The findings reveal that images with higher complexity induce longer and more variable speech onsets, and gaze patterns show significant diversity correlated with these delays. These behavioral signals are crucial indicators of cognitive load and scene ambiguity, yet current multimodal AI models, including prominent pretrained encoders like CLIP and ViT, only weakly capture this variability. Using a similarity prediction approach, the study demonstrates that models' visual features have limited correlation with human signals, highlighting a gap in understanding and modeling human cognition. This underscores the importance of integrating human behavioral cues into training paradigms to develop more natural and diverse language generation systems. The work opens avenues for enhancing AI's alignment with human perception and cognition, fostering more intuitive human-machine interactions. Future directions involve enriching models with additional cognitive signals, exploring cross-linguistic and complex scene scenarios, and employing reinforcement learning to better mimic human variability. Overall, this study advances the understanding of visuo-linguistic behavior and provides practical insights for designing more human-like multimodal AI systems.
Deep Analysis
Background
多模态学习的发展经历了从单一模态到融合多模态信息的演变。早期如ImageNet和BERT主要关注视觉和文本的单模态理解，近年来，模型如VisualBERT、CLIP等通过跨模态对齐实现了更强的表现能力。尽管如此，关于人类在视觉描述中的行为信号（如眼动、语音起始）在模型中的应用仍不足，相关研究多集中在模型性能提升，缺乏对认知偏差的系统分析。已有研究如Sugano和Bulling（2016）利用眼动数据增强图像描述，但未深入分析信号变异的机制。本研究基于Dutch语料库，结合眼动追踪与语音数据，首次系统量化描述行为的多模态变异，填补了认知行为信号在多模态学习中的空白。
Core Problem
当前多模态模型多忽视人类在视觉描述中的行为信号，导致生成的描述缺乏自然性和多样性。模型在捕获视觉复杂性与认知负荷方面表现不足，无法反映人类描述的变异性。如何量化这些信号的变异，并让模型理解其背后的认知机制，成为亟待解决的问题。特别是在多模态融合中，缺乏有效指标衡量模型对人类行为偏差的捕获能力，限制了模型的自然交互能力。
Innovation
本研究提出结合眼动追踪和语音起始时间的多模态信号变异量化框架，创新性地使用预训练视觉编码器（如CLIP、ViT）进行变异预测。引入相似度预测方法，量化模型对视觉-语言变异的敏感度，首次系统分析模型在捕获认知偏差方面的局限。研究强调引入人类行为信号，推动多模态模型更贴近人类认知，提升自然交互的真实性。
Methodology
- �� 数据采集：利用Dutch图像描述语料库（DIDEC），结合眼动追踪和语音数据。
- �� 信号提取：将眼动数据转化为注视区域掩码，利用SAM模型识别注视目标；提取语音起始时间和描述内容。
- �� 信号量化：计算描述起始时间的平均值与标准差，分析起点变异；采用BLEU-2指标衡量描述多样性；设计基于IoU的眼动变异距离。
- �� 相关分析：使用Spearman相关系数，分析不同信号变异间的关系。
- �� 预测模型：利用CLIP和ViT的特征向量，采用相似度预测方法，评估模型对人类变异的捕获能力。
Experiments
实验采用DIDEC语料库，包含307张真实场景图像，45名参与者描述，平均每图15次。通过提取语音起始时间、描述起点、眼动掩码，量化信号变异。对比不同模型的视觉特征（CLIP、ViT）与人类变异指标的相关性，验证模型捕获能力。还进行消融实验，分析不同信号对模型预测性能的贡献。指标包括Spearman相关系数、BLEU-2变异分数等，确保结果的统计显著性。
Results
人类描述的语音起始时间平均为3.42秒，变异范围达25.37秒，明显显示个体差异。描述起点变异与描述多样性呈中等负相关（ρ=-0.516），说明起点偏离会增加描述差异。眼动变异与语音起始时间正相关（ρ=0.455），表明注视变化伴随描述延迟。预训练模型的视觉特征与人类变异的相关性最高约为0.4，显示模型对认知偏差的理解有限。这些结果强调了引入人类信号的重要性，并指出模型在模拟复杂认知行为方面仍有提升空间。
Applications
该研究可应用于提升自动图像描述的自然性与多样性，增强人机交互的自然感。未来可结合认知指标优化模型，适应不同用户的行为偏好，广泛应用于智能助手、辅助技术等场景。长远来看，推动多模态模型理解人类认知机制，实现更智能、更贴近人类的交互系统。
Limitations & Outlook
模型对视觉复杂性偏差的捕获能力有限，未充分考虑个体差异和认知状态。实验仅基于Dutch语料，跨语言适应性不足。信号同步与多模态融合仍需优化，复杂场景中的表现尚未验证。未来需引入更多认知指标，提升模型对人类行为的理解能力。
Plain Language Accessible to non-experts
想象你在厨房做饭。每次准备食材、调味料都像是在观察图片，心里想着怎么搭配。你会注意到不同的食材放置位置（类似眼动），以及你开始准备的时间（像语音起始）。有的人喜欢先切菜，有的人先调味，这就像描述的起点不同。每次做饭，菜的味道和摆盘也会不同，反映出你的习惯和心情。这个过程就像人们描述图片时，注意力、用词和表达方式都不一样。研究发现，这些差异和图片的复杂程度有关，就像厨房里不同的食材难度不同。模型就像一个厨师，试图模仿这些变化，但还不够聪明，不能完全理解每个人的偏好和心情。未来，如果厨师能更好地理解这些细节，就能做出更符合每个人口味的菜肴。
ELI14 Explained like you're 14
想象你和朋友一起玩拼图游戏。每次看到拼图，你的反应时间、注意力集中点和拼图顺序都不一样。有些拼图很简单，你一眼就能找到拼块，开始拼得很快；有些拼图复杂，你会花更多时间观察、思考。每个人的拼图习惯也不同，有的人喜欢从角落开始，有的人喜欢从中心拼起。这就像人们描述图片时，起点、用词和描述方式都不同。这些差异其实和图片的复杂程度有关，就像拼图难度影响你的反应时间。科学家们用眼动追踪和语音记录，试图理解这些差异背后的原因。虽然现在的电脑模型也能描述图片，但还不能完全模仿人类的多样性。未来，如果模型能更好理解这些细节，就能像人一样，讲出更自然、更丰富的描述，让我们和机器的交流更贴近真实人类的感觉！
Abstract
There is an intricate relation between the properties of an image and how humans behave while describing the image. This behavior shows ample variation, as manifested in human signals such as eye movements and when humans start to describe the image. Despite the value of such signals of visuo-linguistic variation, they are virtually disregarded in the training of current pretrained models, which motivates further investigation. Using a corpus of Dutch image descriptions with concurrently collected eye-tracking data, we explore the nature of the variation in visuo-linguistic signals, and find that they correlate with each other. Given this result, we hypothesize that variation stems partly from the properties of the images, and explore whether image representations encoded by pretrained vision encoders can capture such variation. Our results indicate that pretrained models do so to a weak-to-moderate degree, suggesting that the models lack biases about what makes a stimulus complex for humans and what leads to variations in human outputs.