Natural-Language Temporal Grounding in Hour-Long Videos is a Search Problem: A Benchmark and Empirical Decomposition
ExtremeWhenBench benchmark reveals that natural-language temporal grounding in long videos is a search problem; a retrieve-then-ground hybrid improves performance by 6.7x.
Key Findings
Methodology
The study introduces ExtremeWhenBench, an open benchmark for natural-language temporal grounding in long videos. The methodology involves a retrieve-then-ground decomposition, first retrieving long temporal context conditioned on the query, then localizing within a short window. This approach parallels retrieve-then-read in open-domain QA.
Key Results
- The retrieve-then-ground hybrid method improves performance by 6.7x over monolithic Video-LLM on long videos, achieving an mIoU of 0.354.
- On the Charades-STA benchmark, the CLIP retrieval baseline achieves an mIoU of 0.332, while on long videos, it achieves 0.269, outperforming all open Video-LLMs.
- Qwen3.5-9B's mIoU on long videos rises from 0.022 to 0.110, indicating context coverage as the bottleneck.
Significance
This research significantly advances the field of natural-language temporal grounding in long videos. By decomposing the problem into search and localize stages, it highlights the importance of search in long videos. This finding has substantial implications for academia and industry, especially in applications requiring long video content processing.
Technical Contribution
Technical contributions include developing the first open benchmark for long video natural-language temporal grounding, ExtremeWhenBench, and proposing the retrieve-then-ground decomposition. This method effectively addresses the search bottleneck in long videos, offering new engineering possibilities.
Novelty
This is the first systematic study of natural-language temporal grounding in long videos. Unlike existing short video benchmarks, this study provides open-form queries and public access for long videos, significantly expanding the research scope.
Limitations
- The current method relies on a single VLM for 1fps caption stream generation, which may introduce bias.
- The benchmark inherits the genre distribution of LVBench, which may not apply to egocentric or surveillance content.
Future Work
Future work could explore stronger temporally-aware retrievers to improve the mIoU ceiling of the retrieve-then-ground hybrid. Additionally, research could be extended to different types of video content.
AI Executive Summary
Natural-language temporal grounding in long videos is a complex problem that existing short video benchmarks cannot effectively address. The study introduces ExtremeWhenBench, the first open benchmark for long videos, comprising 2,273 queries and 194 videos. By employing a retrieve-then-ground decomposition, the study highlights the importance of search in long videos. Experimental results show that the retrieve-then-ground hybrid method improves performance by 6.7x over monolithic Video-LLM on long videos. This research offers a new perspective for processing long video content, with significant academic and practical implications. Future work could explore stronger temporally-aware retrievers to further enhance performance.
Deep Analysis
Background
With the growth of video content, natural-language temporal grounding has become an important research area. Existing short video benchmarks, such as Charades-STA and ActivityNet, cannot effectively handle complex queries in long videos.
Core Problem
Natural-language temporal grounding in long videos faces dual challenges of search and recognition. Existing methods often fail in long videos mainly due to the large search space.
Innovation
The study introduces ExtremeWhenBench, the first open benchmark for long videos. By employing a retrieve-then-ground decomposition, the study effectively addresses the search bottleneck in long videos.
Methodology
- �� Use Qwen3-VL-8B to generate 1fps caption stream
- �� Perform visual boundary verification with GPT-5.1
- �� Conduct query-conditioned retrieval using CLIP
- �� Localize within a short window
Experiments
Experiments use the ExtremeWhenBench benchmark, comprising 2,273 queries and 194 videos. Baselines include CLIP and various Video-LLMs, with mIoU as the evaluation metric.
Results
The retrieve-then-ground hybrid method achieves an mIoU of 0.354 on long videos, improving performance by 6.7x over monolithic Video-LLM. The CLIP retrieval baseline outperforms all open Video-LLMs on long videos.
Applications
The method can be used for automatic annotation and analysis of long video content, particularly in education and media industries.
Limitations & Outlook
The method relies on a single VLM for caption generation, which may introduce bias. The benchmark's genre distribution may not apply to all types of video content.
Plain Language Accessible to non-experts
Imagine you're in a massive library looking for a book. Traditional methods are like checking each shelf one by one, while the new method first uses keywords to find likely shelves, then searches those shelves carefully. This approach significantly reduces search time, especially when there are many shelves.
ELI14 Explained like you're 14
Imagine playing a huge game of hide and seek. You need to find a friend, but they could be hiding in a massive park. You can first use binoculars to find possible areas, then search those areas carefully. This method helps you find your friend faster!
Glossary
ExtremeWhenBench
An open benchmark for natural-language temporal grounding in long videos, comprising 2,273 queries and 194 videos.
Used to evaluate natural-language temporal grounding performance in long videos.
Video-LLM
Models used for processing video and natural language, capable of understanding and generating descriptions of video content.
Various Video-LLMs were evaluated in the study.
CLIP
A model for matching images and text, trained through contrastive learning.
Used as a baseline for query-conditioned retrieval in long videos.
mIoU
A metric for evaluating model localization accuracy, calculating the intersection-over-union of predicted and true boundaries.
Used to evaluate different methods on long videos.
Retrieve-then-ground
A method that decomposes the long video temporal grounding problem into search and localization stages.
The core methodology proposed in the study.
Open Questions Unanswered questions from this research
- 1 How to effectively apply the retrieve-then-ground method to different types of video content?
- 2 How to enhance the temporal awareness of retrievers to further improve performance?
Applications
Immediate Applications
Educational Video Analysis
Can automatically annotate long educational videos, improving teaching efficiency.
Long-term Vision
Media Content Management
Helps media companies manage and retrieve long video content more effectively.
Abstract
Temporal grounding--returning the interval $[t_s, t_e]$ for a natural-language query over a video--is the language interface to long-form video, yet has been studied on short videos; the dynamics of hour-scale natural-language grounding remain underexplored. We take the position that at hour-scale, the binding constraint is search, not recognition: Video-LLMs are bottlenecked not by localizing a nearby event, but--given a natural-language query--by searching for the relevant region of a long video. To test this, we release ExtremeWhenBench, the first open hour-scale grounding benchmark (2,273 queries over 194 videos, mean 75.7 min, max 9 hr) with an open-form query distribution. Every open Video-LLM collapses while a frame-level retrieval baseline outperforms them; a failure taxonomy attributes 85% of failures to search; and a retrieve-then-ground hybrid recovers 6.7x over the monolithic Video-LLM--mirroring retrieve-then-read in open-domain QA.