Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
Using Orthogonal Matching Pursuit for frame selection in long videos, improving by 6.9 points.
Key Findings
Methodology
The study employs a controlled variable method, fixing frame scorer, prompt boundary, resolution strategy, and answering model, altering selection, spatial compression, and reinvestment decisions one at a time. Orthogonal Matching Pursuit is used for frame selection, testing six training-free selection rules, three long-video benchmarks, and two answering models.
Key Results
- On LongVideoBench's 3600-second videos, selecting 8 relevant frames outperformed 16 uniformly spaced frames by 6.9 points.
- Compressing each frame's spatial budget to half resulted in at most a 0.44-point accuracy drop.
- Reinvesting saved tokens on more compressed frames improved accuracy by 2 to 3 points.
Significance
The study highlights the importance of frame selection in long-video language models, showing that selecting relevant frames is more effective than increasing frame count. This finding is significant for video processing and multimodal learning, challenging traditional uniform sampling methods.
Technical Contribution
The study systematically analyzes the impact of selection, compression, and reinvestment on model performance in long videos, demonstrating the effectiveness of Orthogonal Matching Pursuit for frame selection with detailed experimental support.
Novelty
This study is the first to apply Orthogonal Matching Pursuit for frame selection in long videos, showing its effectiveness without altering other parameters compared to purpose-built selectors.
Limitations
- The study does not account for the computational cost of frame decoding and encoding.
- Experiments are conducted under fixed scorer and answering models, potentially limiting generalizability.
Future Work
Future research could explore frame selection effects under different scorers and answering models, considering the computational cost of frame decoding and encoding.
AI Executive Summary
Long-video language models cannot process every frame, making frame selection crucial. Existing methods often alter multiple factors, complicating comparisons. This study uses a controlled variable method, fixing frame scorer, prompt boundary, resolution strategy, and answering model, altering selection, compression, and reinvestment decisions one at a time. Results show that selecting relevant frames is more effective than increasing frame count. Orthogonal Matching Pursuit matches purpose-built selectors without altering other parameters. Compressing each frame's spatial budget has minimal impact on accuracy, while reinvesting saved tokens on more compressed frames significantly improves accuracy. The study highlights the importance of frame selection in long-video language models, with significant implications for video processing and multimodal learning.
The study employs six training-free selection rules, three long-video benchmarks, and two answering models. Results show that selecting relevant frames is more effective than increasing frame count. Orthogonal Matching Pursuit matches purpose-built selectors without altering other parameters. Compressing each frame's spatial budget has minimal impact on accuracy, while reinvesting saved tokens on more compressed frames significantly improves accuracy.
The study highlights the importance of frame selection in long-video language models, with significant implications for video processing and multimodal learning. Future research could explore frame selection effects under different scorers and answering models, considering the computational cost of frame decoding and encoding.
Deep Analysis
Background
Long-video language models typically cannot process every frame, making frame selection crucial. Traditional methods often use uniform sampling, which does not consider frame relevance and importance. Recent research has proposed various query-based selection methods, but these often alter multiple factors, complicating fair comparisons.
Core Problem
Selecting which frames to retain in long videos is a key issue. Traditional uniform sampling methods fail to fully utilize frame relevance and importance, limiting model performance. How to select the most useful frames under fixed scorer and answering models is an urgent problem.
Innovation
The study systematically analyzes the impact of selection, compression, and reinvestment on model performance in long videos. Using Orthogonal Matching Pursuit for frame selection, it shows its effectiveness without altering other parameters compared to purpose-built selectors.
Methodology
- �� Fix frame scorer, prompt boundary, resolution strategy, and answering model
- �� Alter selection, compression, and reinvestment decisions one at a time
- �� Use Orthogonal Matching Pursuit for frame selection
- �� Test six training-free selection rules, three long-video benchmarks, and two answering models
Experiments
Experiments were conducted on LongVideoBench, Video-MME, and LVBench benchmarks, using Qwen3-VL-8B as the primary answering model. Six selection rules were compared, analyzing the impact of selection, compression, and reinvestment on model performance.
Results
Selecting relevant frames is more effective than increasing frame count. Orthogonal Matching Pursuit matches purpose-built selectors without altering other parameters. Compressing each frame's spatial budget has minimal impact on accuracy, while reinvesting saved tokens on more compressed frames significantly improves accuracy.
Applications
The study's findings can optimize frame selection strategies in long-video language models, improving model performance and reducing computational costs.
Limitations & Outlook
The study does not account for the computational cost of frame decoding and encoding. Experiments are conducted under fixed scorer and answering models, potentially limiting generalizability.
Plain Language Accessible to non-experts
Imagine watching a long movie but only being able to choose a few key scenes to understand the whole story. Traditional methods are like taking a snapshot every few minutes, while this study's method is more like selecting the most important scenes based on the plot. By choosing these key scenes, you can better understand the movie's storyline without watching the entire film. It's like studying for an exam by focusing on the most likely topics to be tested rather than reviewing everything.
ELI14 Explained like you're 14
Imagine playing a super long game but only being able to save a few key progress points. Traditional methods are like saving every few minutes, while this study's method is more like choosing the most important save points based on the game's storyline. This way, you can better understand the game's overall story without saving every progress point. It's like choosing the most likely topics to be tested in school rather than reviewing everything. Isn't that cool?
Glossary
Orthogonal Matching Pursuit
A greedy sparse approximation algorithm that selects candidates most correlated with the query.
Used for selecting key frames in long videos.
Visual Token
The basic unit representing video frames, similar to words in text.
Used in video processing and multimodal learning.
Uniform Sampling
A method of selecting frames by evenly extracting a fixed number of frames from the video.
Traditional frame selection method.
Spatial Compression
Reducing the spatial budget of each frame to lower computational costs.
Used to optimize frame selection strategies.
Reinvestment
Using saved tokens for more compressed frames to improve model performance.
Used in optimizing frame selection strategies.
Open Questions Unanswered questions from this research
- 1 How to optimize frame selection strategies under different scorers and answering models?
- 2 How do computational costs of frame decoding and encoding affect model performance?
- 3 How to improve frame selection accuracy without increasing computational costs?
Applications
Immediate Applications
Video Processing Optimization
By optimizing frame selection strategies, improve long-video language model performance and reduce computational costs.
Long-term Vision
Multimodal Learning
By improving frame selection strategies, advance multimodal learning and achieve more efficient video processing.
Abstract
Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.