ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
ScaleLong evaluates long video understanding across timescales, revealing a U-shaped performance curve.
Key Findings
Methodology
ScaleLong embeds questions targeting four hierarchical timescales (Clip, Shot, Event, Story) within the same video content, allowing direct comparison of MLLM performance across timescales. It features 269 long videos, averaging 86 minutes, across 5 main categories and 36 subcategories.
Key Results
- Result 1: Evaluating 23 MLLMs reveals a U-shaped performance curve, with higher accuracy at the shortest and longest timescales, and a dip at intermediate levels.
- Result 2: Increased visual token capacity consistently enhances reasoning across all timescales.
- Result 3: Closed-source models outperform open-source ones across all timescales, with Gemini 2.5 Pro performing best on Clip and Story scales.
Significance
ScaleLong provides a fine-grained, multi-timescale benchmark for long video understanding, addressing the gap in existing benchmarks' timescale design. It offers insights into MLLM performance across different timescales, guiding future model improvements.
Technical Contribution
ScaleLong decouples timescale understanding from content variability by embedding multi-timescale questions within the same video, enabling precise evaluation of MLLM capabilities in understanding temporal structures in long videos.
Novelty
ScaleLong is the first benchmark to embed multi-timescale questions within the same video content, solving the issue of timescale and content variability coupling in previous benchmarks.
Limitations
- Limitation 1: Performance dips at intermediate timescales, indicating challenges in handling moderate-length temporal contexts.
- Limitation 2: Dataset diversity may limit generalization to specific domains.
Future Work
Future work can focus on improving performance at intermediate timescales and exploring how to optimize visual token allocation without increasing computational costs.
AI Executive Summary
Understanding long videos requires models to capture hierarchical temporal information from clips to stories. Existing benchmarks either neglect this multi-scale design or scatter scale-specific questions across different videos, preventing direct comparison of model performance across timescales on the same content. To address this, we introduce ScaleLong, the first benchmark to disentangle these factors by embedding questions targeting four hierarchical timescales—Clip, Shot, Event, and Story—all within the same video content. This within-content multi-timescale questioning design enables direct comparison of model performance across timescales on identical videos. ScaleLong features 269 long videos (avg. 86 min) from 5 main categories and 36 sub-categories, with 4–8 carefully designed questions, including at least one question for each timescale. Evaluating 23 MLLMs reveals a U-shaped performance curve, with higher accuracy at the shortest and longest timescales and a dip at intermediate levels. Furthermore, ablation studies show that increased visual token capacity consistently enhances reasoning across all timescales. ScaleLong offers a fine-grained, multi-timescale benchmark for advancing MLLM capabilities in long-video understanding. The code and dataset are available at https://github.com/multimodal-art-projection/ScaleLong.
Deep Analysis
Background
Long video understanding requires models to capture hierarchical temporal information from short clips to long stories. Existing benchmarks often neglect this multi-timescale design or scatter scale-specific questions across different videos, making it difficult to compare model performance across timescales on the same content.
Core Problem
Existing benchmarks are ill-equipped to assess MLLM capabilities across different timescales, particularly in long videos. This makes it difficult to disentangle a model's true performance at each specific timescale from content-driven adaptations.
Innovation
ScaleLong embeds multi-timescale questions within the same video, decoupling timescale understanding from content variability, providing precise evaluation of MLLM capabilities in understanding temporal structures in long videos.
Methodology
- �� Embed questions targeting four hierarchical timescales (Clip, Shot, Event, Story) within the same video
- �� Design 4-8 questions per video, ensuring at least one question per timescale
- �� Use 269 long videos, averaging 86 minutes, across 5 main categories and 36 subcategories
Experiments
The experiment evaluated 23 MLLMs, including 19 open-source models and 4 proprietary models. Different visual token configurations were used to test model performance across timescales.
Results
Results show models perform better at the shortest and longest timescales, with a dip at intermediate levels. Increased visual token capacity enhances reasoning across all timescales.
Applications
ScaleLong can be used to evaluate and improve MLLM capabilities in long video understanding, especially in applications requiring complex temporal structure capture.
Limitations & Outlook
While ScaleLong provides fine-grained evaluation, performance dips at intermediate timescales indicate challenges in handling moderate-length temporal contexts. Future work should focus on improving performance at these timescales.
Plain Language Accessible to non-experts
Imagine you're watching a movie with many scenes and storylines. Understanding this movie is like a puzzle game, where you need to connect each small piece (like a scene) with the whole story (like the entire movie). ScaleLong is like a compass, helping us evaluate if machines can understand these complex movie plots like humans. By asking different time-length questions within the same movie, we can see where machines do well and where they need improvement. Just like in a puzzle game, some pieces are easy to fit, while others require more time and effort.
ELI14 Explained like you're 14
Imagine you're playing a super complex game with many levels and missions. Each level has different time limits, some just a few seconds, others maybe hours. ScaleLong is like a game review system, helping us see if computers can be as smart as you in completing these tasks. By setting different time challenges in the same game, we can see where computers do well and where they need more practice. Just like you in a game, some levels are easy to pass, while others need more strategy and skill.
Glossary
Multimodal Large Language Models (MLLMs)
Models capable of processing multiple input forms such as text, images, and videos.
Used in the paper to evaluate their performance in long video understanding.
Timescale
Refers to different lengths of time segments in a video, such as Clip, Shot, Event, and Story.
Used to design questions to evaluate model understanding across different time lengths.
Visual Token
The basic unit representing visual information in a video.
In experiments, the number of visual tokens is adjusted to test model performance.
Ablation Study
Research method involving the removal or modification of certain components to study their impact on overall performance.
Used to analyze the impact of increasing visual token capacity on model performance.
U-shaped Performance Curve
A performance trend where accuracy is higher at extreme timescales and dips at intermediate scales.
Observed in the experimental results as a model performance trend.
Open Questions Unanswered questions from this research
- 1 How to improve MLLM performance at intermediate timescales? Current methods struggle with moderate-length temporal contexts.
- 2 How to optimize visual token allocation without increasing computational costs?
- 3 How to enhance dataset diversity to improve generalization capabilities?
Applications
Immediate Applications
Video Analysis
Can be used to evaluate and improve video analysis systems, especially in applications requiring complex temporal structure capture.
Long-term Vision
Intelligent Video Understanding
Advances the development of intelligent video understanding technologies, improving machine performance in complex video scenarios.
Abstract
Although long-video understanding demands that models capture hierarchical temporal information -- from clip (seconds) and shot (tens of seconds) to event (minutes) and story (hours) -- existing benchmarks either neglect this multi-scale design or scatter scale-specific questions across different videos, preventing direct comparison of model performance across timescales on the same content. To address this, we introduce ScaleLong, the first benchmark to disentangle these factors by embedding questions targeting four hierarchical timescales -- clip (seconds), shot (tens of seconds), event (minutes), and story (hours) -- all within the same video content. This within-content multi-timescale questioning design enables direct comparison of model performance across timescales on identical videos. ScaleLong features 269 long videos (avg.\ 86\,min) from 5 main categories and 36 sub-categories, with 4--8 carefully designed questions, including at least one question for each timescale. Evaluating 23 MLLMs reveals a U-shaped performance curve, with higher accuracy at the shortest and longest timescales and a dip at intermediate levels. Furthermore, ablation studies show that increased visual token capacity consistently enhances reasoning across all timescales. ScaleLong offers a fine-grained, multi-timescale benchmark for advancing MLLM capabilities in long-video understanding. The code and dataset are available https://github.com/multimodal-art-projection/ScaleLong.