A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering

The evolution of vision-language models (VLMs) from CNN-RNN architectures to large pre-trained models like Video-LLaVA and InternVideo has significantly advanced video understanding. These models leverage extensive pretraining to perform zero-shot reasoning across visual and textual modalities, enabling tasks like VideoQA. However, processing long videos remains computationally challenging due to the sheer number of frames and limited context windows. Existing frame sampling strategies, such as uniform sampling, often fail to capture query-relevant content, especially for complex questions requiring temporal reasoning. Lightweight similarity models like CLIP are efficient but lack semantic depth, leading to inaccurate relevance scores. Deep VLM analysis offers accuracy but is computationally expensive, limiting practical deployment. Consequently, there is a pressing need for methods that balance semantic depth with efficiency, especially for real-time applications.

The core challenge in VideoQA is selecting a small, highly relevant subset of frames from long videos for accurate reasoning. Shallow models like CLIP cannot reliably capture complex semantic relationships, resulting in poor frame relevance ranking. Deep analysis with large VLMs, while accurate, is prohibitively costly when applied to all frames. This creates a bottleneck: how to achieve deep understanding without excessive computation? Existing solutions either sacrifice semantic depth or computational feasibility. Addressing this requires a method that adaptively identifies key content, efficiently allocates deep analysis resources, and iteratively refines frame selection to handle diverse video lengths and query complexities.

A.I.R. introduces a novel, training-free framework that combines statistical event detection with iterative deep reasoning. Its key innovations include: 1) Adaptive Initial Sampling using GMM to identify potential event regions based on query similarity, ensuring targeted coverage; 2) Interval Potential Ranking that assesses temporal segments collectively rather than individual frames, improving robustness; 3) Deep reasoning-based validation via large VLMs, which confirms the relevance of candidate frames; 4) An early stopping mechanism to prevent unnecessary computation; 5) Localized Density Sampling that explores nearby frames around validated content, refining the selection iteratively. These components work synergistically to deliver high-precision frame selection with minimal resource expenditure, enabling scalable VideoQA.

• �� Initial sampling: uniformly sample n frames, compute query-frame similarity via CLIP, form sparse vector S.
• �� Adaptive event detection: fit GMM to S, determine threshold T, segment video into events.
• �� Event refinement: merge short events, prune insignificant ones, finalize event set.
• �� Event-wise sampling: select peak similarity frames proportionally within each event.
• �� Interval potential ranking: partition video into intervals based on sampled frames, compute potential scores considering relevance, complexity, and length.
• �� Deep validation: use reasoning VLM with prompts to score and justify frame relevance, classify as positive/neutral/negative.
• �� Early stop: accumulate validated frames, halt if budget met.
• �� Localized density sampling: around validated frames, sample additional frames with exponential stride, update similarity scores, and feed into next iteration.
• �� Repeat until budget exhausted or max iterations reached, then perform final inference with selected frames.

The authors evaluate A.I.R. on datasets including TVQA, TVR, and ActivityNet, comparing against baseline uniform sampling and shallow similarity methods. Metrics include accuracy, inference time, and resource consumption. They test multiple VLM architectures, such as InternVL-3-8B and QwenVL, performing ablation studies to assess each component's impact. Hyperparameters like the similarity threshold γ and maximum iteration count are tuned for robustness. Results show consistent accuracy improvements, significant reductions in computational costs, and high adaptability across models. Ablation confirms the importance of adaptive sampling and iterative validation, demonstrating the framework's effectiveness in diverse scenarios.

A.I.R. achieves an average accuracy increase of 8.5% over baseline methods on VideoQA benchmarks, with a 70% reduction in total inference time. In complex long videos, it maintains high relevance detection and reasoning accuracy while saving computational resources. Ablation studies reveal that adaptive event detection improves recall to 92%, and iterative deep validation reduces false positives. The framework's compatibility with various VLMs underscores its versatility, and experiments confirm that targeted, iterative analysis outperforms single-pass deep analysis, especially in resource-constrained settings.