Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA

Video understanding has evolved from early hierarchical models (Allen & Ferguson, 1994) to modern deep learning approaches integrating language modalities. Datasets like Xiao et al. (2021) introduced long-form VideoQA, emphasizing causal and temporal reasoning. Despite progress, processing lengthy videos remains computationally demanding, especially with full-frame methods. Recent advances utilize large pre-trained models (GPT-4, CLIP) for multimodal reasoning, but their application to hours-long videos faces efficiency bottlenecks. Keyframe selection strategies have emerged as a solution, aiming to extract minimal yet informative frames. Existing methods often rely on uniform sampling or simple heuristics, which fail to capture dynamic scene changes or relevant content, limiting accuracy and scalability. This background underscores the need for more intelligent, hierarchical filtering mechanisms that can adaptively identify critical content in long videos, enabling scalable, real-time applications.

The core challenge in long-video QA is balancing the need for comprehensive understanding with computational feasibility. Uniform sampling leads to redundant data processing, while full-frame analysis is prohibitively expensive. Existing methods lack effective hierarchical filtering, resulting in either low accuracy or high computational costs. The problem becomes more acute with videos spanning hours, where irrelevant content dominates. Efficiently identifying and describing only the most relevant frames without training on specific videos remains an open question. Addressing this bottleneck is crucial for deploying scalable, real-time video understanding systems capable of handling diverse, lengthy content.

LVNet introduces a hierarchical keyframe selection framework that combines scene clustering, keyword relevance, and visual templating. The scene clustering (TSC) reduces frames from 900 to ~390 by grouping similar scenes. The coarse filtering (CKD) leverages a dual-encoder CLIP model to match keywords derived from questions, selecting 32 frames with high relevance. The fine filtering (FKD) uses visual templating to assemble frames into composite images, enabling detailed reasoning and further filtering down to 12 keyframes. This layered approach ensures minimal data input while maintaining high semantic fidelity. Unlike prior uniform sampling or single-stage methods, LVNet's multi-stage filtering adapts dynamically to scene content and query relevance, significantly reducing computational load without sacrificing accuracy.

• �� Input: Long video and natural language question. • Extract 900 frames using ResNet-18. • Scene clustering (TSC): Cluster frames into scenes, sample representative frames to reduce to ~390. • Coarse filtering (CKD): Use CLIP to match question keywords with frames, select 32 frames based on relevance and confidence scores. • Visual templating (FKD): Group top frames into sets, create composite images, and use VLM to select the most relevant frames, reducing to 12. • Description: Generate natural language descriptions of selected frames via VLM. • Answer generation: Feed descriptions and question into GPT-4 to produce answer. This layered process balances efficiency and accuracy, progressively narrowing down from hundreds to a dozen keyframes.

The evaluation uses datasets like EgoSchema, NExT-QA, IntentQA, and VideoMME, testing the model's ability to answer questions with minimal frames. The experiments compare LVNet against uniform sampling, single-stage models, and other keyframe selection methods, measuring accuracy and computational cost. Hyperparameters include 900 initial frames, 12 keyframes for inference, and the use of pre-trained ResNet-18, CLIP-B/16, and GPT-4. Ablation studies analyze the contribution of each HKS component, confirming that hierarchical filtering improves both efficiency and accuracy. Results show LVNet surpasses existing methods by 7-10% in accuracy while processing only 12 frames, demonstrating superior scalability and robustness across diverse video lengths and content complexities.

LVNet achieves 61.1% accuracy on EgoSchema, 72.9% on NExT-QA, and 71.7% on IntentQA using only 12 frames, outperforming comparable models that process more captions. On hour-long videos (VideoMME), it maintains 53.9% accuracy, surpassing single-stage methods by nearly 10%. Ablation results confirm each filtering stage's importance, with combined keyword and temporal strategies yielding best performance. The method significantly reduces computational costs, over 10x less inference expense compared to direct GPT-4 processing of full videos, making long-video QA feasible at scale.

This approach enables efficient, scalable long-video understanding for applications like automated content moderation, video summarization, and interactive AI assistants. It reduces hardware requirements and inference costs, making real-time deployment practical. The training-free design allows easy integration into existing pipelines, broadening accessibility. Long-term, it can facilitate comprehensive video analytics in surveillance, education, and entertainment, transforming how large-scale video data is processed and understood.

While effective, LVNet's performance may decline with videos featuring rapid scene changes or complex backgrounds where scene clustering and keyword matching are less reliable. Dependence on pre-trained models limits domain adaptation. Handling ultra-long videos (multiple hours) still incurs some computational overhead, requiring further optimization. Future work should focus on robustness in diverse scenarios, real-time processing, and adaptive filtering strategies.