TempCloze: Can Video-LLMs Identify the Missing Middle?
TempCloze evaluates visual temporal reasoning by identifying the missing middle segment from video clips, revealing that alignment remains the main bottleneck.
Key Findings
Methodology
TempCloze constructs a dataset of 1521 videos from seven sources, emphasizing long-take and egocentric videos. It introduces three types of distractors—Semantic, Alignment, and Progression—each designed to challenge different aspects of temporal reasoning. Models are tasked with selecting the correct middle clip from four candidates, which share scenes and objects with the original, thus minimizing appearance cues. The dataset employs optical flow filtering to exclude static videos and uses large language models (GPT-3) for caption filtering. Evaluation across 31 models reveals that while models perform reasonably well on semantic content and event progression, their ability to accurately align the temporal position remains weak, with average alignment accuracy around 48%. Error and behavioral analyses further show that models are highly sensitive to candidate order, context direction, visible span, and frame density, with alignment consistently being the most challenging dimension.
Key Results
- Across 1521 videos, the best models achieve 70.81% full understanding (all three dimensions correct), but the alignment accuracy is only 48.13%, far below human performance of 98%. Open-source models average 34%, with the top reaching 75.94%. Alignment errors dominate, especially with expanded or reversed distractors. Models are highly sensitive to candidate order, leading to unstable predictions. Increasing test-time attempts improves accuracy but does not resolve the core alignment bottleneck. These findings highlight that current models excel at content and event progression but struggle with precise temporal placement, indicating a critical area for future improvement.
Significance
This work introduces TempCloze, a novel benchmark that directly assesses visual temporal reasoning without language mediation, addressing a key limitation of existing benchmarks like TempCompass and TVBench. By focusing on pure visual cues and multi-dimensional distractors, it provides a more accurate measure of a model’s understanding of event timing and sequence. The results reveal that despite progress, models still face significant challenges in temporal alignment, which is crucial for applications like video summarization, surveillance, and autonomous systems. The benchmark thus guides future research towards developing video-native temporal modeling mechanisms, fostering advancements in AI’s ability to interpret complex dynamic scenes with high temporal fidelity. It also establishes a standardized evaluation framework that can accelerate progress in the field.
Technical Contribution
The paper presents a comprehensive multi-dimensional distractor-based benchmark, combining semantic, alignment, and progression challenges, to evaluate models’ temporal reasoning. It innovates by using same-source scene sharing to reduce appearance cues, optical flow filtering for quality control, and large-scale evaluation of diverse models. The analysis uncovers that alignment remains the primary bottleneck, guiding future model design to focus on temporal boundary detection and multi-directional reasoning. The methodology integrates controlled distractor variants and stability tests, providing a detailed diagnostic of model weaknesses. This approach advances the state-of-the-art in visual temporal reasoning evaluation, offering a rigorous, interpretable, and scalable benchmark for future research.
Novelty
TempCloze is the first benchmark to systematically evaluate pure visual temporal reasoning through a multi-dimensional distractor framework, emphasizing the importance of precise time alignment. Unlike prior benchmarks that rely heavily on language or simple event ordering, it introduces complex distractors that challenge models’ ability to discern subtle temporal boundaries, event unfolding, and sequence reversals. The use of same-source scene sharing minimizes appearance cues, forcing models to rely on temporal cues. This comprehensive, multi-faceted evaluation fills a critical gap in video understanding research, providing a new standard for assessing models’ deep temporal comprehension beyond surface-level content recognition.
Limitations
- The benchmark focuses solely on missing middle identification, not on open-ended generation, narrative understanding, or audio-grounded inference, limiting its scope for comprehensive video understanding evaluation.
- All distractors are from the same source videos, which may underestimate models’ generalization ability in real-world scenarios with diverse scenes.
- Model performance is sensitive to candidate order, sampling density, and context direction, indicating that current models lack robustness and stability in temporal reasoning.
Future Work
Future research should explore video-native architectures that explicitly model bidirectional temporal dependencies, such as transformer-based spatiotemporal modules. Extending the benchmark to longer, more complex videos, incorporating multi-modal cues like audio and text, will provide a more holistic evaluation. Developing self-supervised learning strategies to improve temporal boundary detection and event sequencing is also promising. Additionally, designing models with better stability against candidate permutations and sampling variations will enhance robustness. Ultimately, integrating these advances could lead to AI systems capable of human-like understanding of complex dynamic scenes in real-world applications.
AI Executive Summary
Understanding the temporal structure of videos remains a fundamental challenge in AI. Traditional benchmarks often rely on language-based questions, which can be exploited through superficial cues, thus failing to accurately measure a model’s true understanding of event timing and sequence. To address this, TempCloze introduces a novel, purely visual benchmark that requires models to identify a missing middle segment from a set of candidates, carefully designed to minimize appearance cues and emphasize temporal reasoning.
Constructed from 1521 videos across diverse sources, the dataset employs multi-dimensional distractors—semantic, alignment, and progression—to probe different facets of temporal understanding. The evaluation of 31 models reveals a consistent pattern: while models can recognize plausible event content and local event progression, their ability to precisely align the event in time remains significantly limited, with alignment accuracy around 48%. This bottleneck persists even when increasing test-time attempts, indicating a fundamental challenge in temporal boundary detection.
Error analysis shows models are highly sensitive to candidate order and context direction, often relying heavily on endpoint cues. Behaviorally, models depend more on the beginning context, and longer frame spans can dilute critical temporal signals. These insights highlight the need for video-native temporal modeling mechanisms that can explicitly capture bidirectional dependencies and event boundaries.
The significance of this work lies in providing a rigorous, interpretable benchmark that directly assesses visual temporal reasoning, guiding future research toward models capable of understanding complex dynamic scenes with human-level accuracy. By focusing on pure visual cues and multi-dimensional challenges, TempCloze sets a new standard for evaluating and advancing AI’s temporal understanding in videos.
Deep Analysis
Background
Video understanding技术已从动作识别逐步演进到复杂事件推理，代表性工作如TGIF-QA、ActivityNet-QA、LongVideoBench，逐步拓展对长时动态和细粒度事件的理解能力。然而，现有基准多依赖语言描述，容易被表面特征或语言偏见所影响，限制了模型对深层时间关系的理解。近年来，TempCompass和TVBench引入事件排序和定位任务，试图强化时间结构理解，但在复杂场景中的时间对齐仍存瓶颈。随着多模态模型的发展，评估标准亟需突破传统语言中介，直观考察视觉时间推理能力，成为研究热点。
Core Problem
核心问题在于，尽管模型在事件内容和演进方面表现良好，但在精确对齐事件时间点时，错误率高达50%以上。这主要源于模型对时间边界的敏感性不足，难以区分相似场景中的微妙时间差异。此外，模型对候选顺序和上下文方向的依赖，使得推理结果不稳定。解决这一问题对于实现更智能的视频理解系统至关重要，尤其在监控、虚拟现实等应用中，时间的准确把握直接影响系统性能。
Innovation
本研究的创新点包括：1）提出多维干扰设计，涵盖事件应发生、时间应匹配、事件应展开三大维度，增强模型对时间关系的敏感性；2）利用长时段和自我中心视频，减少外观线索干扰，强化时间推理；3）引入光流算法筛除静态视频，确保数据质量；4）系统评估多模型，揭示对齐为主要瓶颈。不同于传统只关注事件内容的基准，TempCloze强调时间边界的精确识别，为模型原生时间建模提供了新平台。这些创新推动模型在复杂动态场景中的时间理解能力。
Methodology
- �� 数据采集：从7个公开源筛选1521个长时段或自我中心视频，过滤静态和低质量视频。• 视频预处理：采用光流算法筛除静态场景，采样中间段，确保上下文完整。• 干扰设计：构建三维干扰（语义、对齐、演进），每维度包含多种变体（如扩展、倒转、重排、重复），以测试模型对时间关系的敏感性。• 模型评估：使用21个开源和10个专有模型，统一采样16帧，测试在不同干扰下的表现。• 误差分析：分析模型在不同干扰类型中的错误分布，识别主要瓶颈。• 行为敏感性：测试候选排序、上下文方向、帧密度和测试尺度对模型稳定性和准确率的影响。
Experiments
采用多源数据集，评估模型在三维干扰下的准确率，统计完整理解（全对）比例。设置不同候选排序和采样密度，分析模型稳定性。引入TempCloze-Mixed和TempCloze-Hard两个子集，检测模型在复杂场景中的表现。通过多轮测试和温度调节，评估模型预测的稳定性和鲁棒性。实验结果显示，模型在语义和演进方面表现较好，但在对齐任务中表现最差，准确率明显低于人类水平。模型对候选顺序敏感，表现出较高的不稳定性，验证了对齐为主要瓶颈。
Results
模型在1521个视频中，最高达70.81%的完整理解率，但在对齐任务中仅48.13%，远低于人类的98%。开源模型平均准确率为34%，最高达75.94%。对齐干扰（如扩展、倒转）错误率最高，模型对候选顺序极敏感。行为分析显示，模型更依赖起点信息，长帧采样反而稀释关键信息。测试尺度提升后，模型性能有所改善，但对齐瓶颈依旧明显。这些数据明确指出，现有模型在时间对齐方面仍需突破，未来应发展视频原生的时间建模机制。
Applications
该基准可用于自动视频内容分析、行为识别和事件定位，特别适合长时动态场景的理解。通过优化模型对时间边界的敏感性，可提升监控、虚拟现实和智能交互系统的性能。未来，结合多模态信息（如声音、文本）将拓展其应用范围，推动自动化视频分析向更高层次发展。
Limitations & Outlook
目前基准主要关注缺失中段识别，未覆盖开放式生成、对话叙事或音频推理场景。候选来自同源视频，虽然减少外观线索，但可能低估模型泛化能力。模型对候选排序和采样密度敏感，表现不稳定。未来需设计更复杂、多模态、多任务的评估体系，以全面衡量模型的时序理解能力。
Plain Language Accessible to non-experts
想象你在看一本故事书，书的前几页和后几页都很清楚，但中间的一段被撕掉了。你需要凭借已知的前后内容，猜出中间发生了什么。这个过程就像TempCloze任务，模型要根据视频的开始和结束片段，猜出中间缺失的内容。为了让猜测更难，研究者设计了各种干扰，比如让模型区分事件是否在正确的时间发生，或者事件是否按正确的顺序展开。模型要理解事件的内容、时间点和发展过程，就像拼图一样，把碎片拼成完整的画面。这个任务帮助我们了解模型是否真正理解了视频中的时间关系，而不是仅仅依赖外观或表面特征。它就像让模型成为一个聪明的侦探，能准确判断事件发生的时间和顺序，而不是只看表面。
ELI14 Explained like you're 14
想象你在看一部电影，电影的前后部分都很清楚，但中间的内容被剪掉了。你的任务是猜出中间发生了什么。你可以根据前后场景猜测，比如看到一场比赛的开始和结束，猜中间可能是比赛进行中。这个游戏看起来简单，但其实很难，因为你要理解事件的时间顺序和发展过程。研究人员设计了类似的游戏，让电脑模型也来猜中间缺失的内容。他们用各种干扰，比如让模型区分事件是否在正确的时间发生，或者事件的顺序是否正确。模型需要像侦探一样，理解视频中的每个细节，判断事件发生的时间和顺序。这个游戏帮助我们知道，电脑是不是真正理解了视频的时间关系，而不是只会看表面。未来，这样的技术可以让电脑更聪明，能更好地理解视频内容，比如自动监控、虚拟现实等都能用到。
Abstract
Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correlations, or language priors. To reduce such shortcuts, we introduce TempCloze, a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs. Given the beginning and ending clips of a video, models must identify the true missing middle from four candidates. TempCloze contains 1,521 carefully filtered videos from seven sources, mainly long-take and egocentric videos. We construct same-source distractors along three dimensions: Semantic asks what event should happen, Alignment probes when it should occur, and Progression tests how it should unfold, while shared scenes and objects reduce appearance cues. Our evaluation of 10 proprietary and 21 open-source Video-LLMs reveals Alignment as the primary bottleneck: models often recognize plausible semantic content and local event progression but struggle with temporal alignment. We further conduct error pattern and behavioral sensitivity analyses on TempCloze-Mixed and TempCloze-Hard with four representative models to examine where errors arise and how candidate order, context direction, visible span, frame density, and test-time scaling influence model choices.