ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QA
ReQuest improves long-video QA accuracy through uncertainty-driven keyframe selection.
Key Findings
Methodology
ReQuest integrates a lightweight question-aware selector, re-thinking routing, and uncertainty-guided adaptive non-maximum suppression. The selector is distilled from MLLM-generated supervision, routing triggers additional inference based on model uncertainty, and suppression adjusts frame spacing according to question difficulty.
Key Results
- On Video-MME, ReQuest improved accuracy from 62.6% to 65.6%, with notable gains in medium and long videos, increasing to 64.1% and 55.8% respectively.
- On MLVU and LongVideoBench, accuracy increased to 73.9% and 60.1%, demonstrating cross-benchmark generalization.
- Through adaptive non-maximum suppression, ReQuest effectively selects informative frames in long videos.
Significance
ReQuest significantly enhances the accuracy and efficiency of long-video QA, addressing the challenge of evidence localization under fixed input token budgets. Its innovative keyframe selection mechanism offers new insights for multimodal large language models in long-video understanding.
Technical Contribution
ReQuest improves long-video QA performance as a plug-and-play method without modifying or fine-tuning the underlying MLLM. Its uncertainty-driven selection mechanism provides new theoretical guarantees and engineering possibilities.
Novelty
ReQuest uniquely combines uncertainty-driven selection strategies with a question-aware selector, significantly improving efficiency and accuracy in long-video QA.
Limitations
- In complex scenarios, the selector may fail to accurately identify keyframes, leading to inference errors.
- Dependence on video length may affect performance in extremely long videos.
Future Work
Future work could explore optimizing selector performance, especially in extremely long videos, and integrating other multimodal information to enhance QA quality.
AI Executive Summary
Long-video QA has been a challenging task for multimodal large language models. Traditional methods struggle with evidence localization under fixed input token budgets, leading to information loss. ReQuest addresses this by introducing an uncertainty-driven keyframe selection mechanism, combining a lightweight question-aware selector and re-thinking routing, significantly improving accuracy and efficiency.
The ReQuest selector is distilled from MLLM-generated supervision, capable of selecting relevant video content based on question intent. Re-thinking routing triggers additional inference when the model is uncertain, ensuring selected frames provide rich semantic information.
Experimental results show that ReQuest achieves significant accuracy improvements across Video-MME, MLVU, and LongVideoBench, particularly excelling in medium and long videos. Its innovative selection mechanism offers new insights for long-video understanding while maintaining competitive computational costs.
Deep Analysis
Background
With the development of multimodal large language models, the video QA field has made significant progress. However, long-video QA remains challenging, especially in evidence localization under fixed input token budgets. Traditional methods often adopt uniform sampling but risk missing critical evidence.
Core Problem
The core problem of long-video QA is effectively locating evidence under limited token budgets. Uniform sampling often fails to capture keyframes, leading to information loss and affecting QA accuracy.
Innovation
ReQuest addresses evidence localization in long-video QA through an uncertainty-driven keyframe selection mechanism, combining a lightweight question-aware selector and re-thinking routing. Its innovation lies in combining uncertainty-driven selection strategies with a question-aware selector, improving efficiency and accuracy.
Methodology
- �� Lightweight Question-Aware Selector: Distilled from MLLM-generated supervision, capable of selecting relevant video content based on question intent.
- �� Re-thinking Routing: Triggers additional inference when the model is uncertain, ensuring selected frames provide rich semantic information.
- �� Adaptive Non-Maximum Suppression: Adjusts frame spacing according to question difficulty, selecting temporally diverse frames.
Experiments
Experiments were conducted on Video-MME, MLVU, and LongVideoBench using LLaVA-Video as the baseline model. Frames were sampled at 1 fps and encoded to obtain fused frame-question representations. The selector was trained to regress toward pseudo targets. Results show ReQuest significantly improved QA accuracy.
Results
ReQuest improved accuracy on Video-MME from 62.6% to 65.6%, and on MLVU and LongVideoBench to 73.9% and 60.1% respectively. Its selection mechanism effectively chose informative frames, demonstrating cross-benchmark generalization.
Applications
ReQuest can be applied in video QA systems, especially suitable for scenarios requiring long-video processing, such as educational video analysis and film content retrieval.
Limitations & Outlook
ReQuest may fail to accurately identify keyframes in complex scenarios, leading to inference errors. Dependence on video length may affect performance in extremely long videos.
Plain Language Accessible to non-experts
Imagine watching a long movie and needing to answer questions about its content. Traditional methods are like taking a snapshot every few minutes, which might miss crucial scenes. ReQuest acts like a smart assistant, selecting the most relevant frames based on the question and re-thinking when uncertain, ensuring you get the right answer.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super long game and need to answer questions about it. Traditional methods are like taking a screenshot every few minutes, which might miss key game scenes. ReQuest is like a smart assistant, selecting the most relevant game frames based on the question and re-thinking when uncertain, ensuring you win the game!
Glossary
ReQuest
A keyframe selection mechanism for long-video QA, combining uncertainty-driven selection strategies with a question-aware selector.
Used in the paper to improve long-video QA accuracy.
MLLM
A model that jointly encodes visual and textual inputs for high-level reasoning.
The foundational model for video QA.
Key Frame Selection
A method for selecting the most relevant frames from a video to improve QA accuracy.
Core mechanism of ReQuest.
Re-thinking Routing
A mechanism that triggers additional inference when the model is uncertain.
Ensures selected frames provide rich semantic information.
Adaptive NMS
A selection strategy that adjusts frame spacing according to question difficulty.
Used to select temporally diverse frames.
Open Questions Unanswered questions from this research
- 1 Optimizing selector performance in extremely long videos remains an open question.
- 2 Exploring integration of other multimodal information to enhance QA quality is needed.
Applications
Immediate Applications
Educational Video Analysis
Improving accuracy of educational video QA systems through keyframe selection.
Long-term Vision
Film Content Retrieval
Achieving efficient film content retrieval through intelligent keyframe selection.
Abstract
Recent multimodal large language models (MLLMs) have substantially advanced video understanding, yet long-form video QA remains challenging under fixed input token budgets, where uniform sampling can be inefficient for evidence localization. We propose ReQuest , an uncertainty-driven, question-adaptive keyframe selection pipeline that aligns question intent with relevant video content through selective computation. ReQuest integrates (i) a lightweight question-aware selector distilled from MLLM-generated supervision, (ii) Re-thinking Routing that triggers additional inference only when the model is uncertain with a length-adaptive criterion, and (iii) uncertainty-guided adaptive non-maximum suppression that selects temporally diverse frames while adjusting spacing based on question difficulty. As a plug-andplay method, ReQuest improves long-video QA without modifying or fine-tuning the underlying MLLM. Experiments on Video-MME, MLVU, and LongVideoBench demonstrate consistent accuracy gains with competitive computational cost, with particularly strong improvements in medium and long video regimes.