A Simple LLM Framework for Long-Range Video Question-Answering

Long video understanding has been a longstanding challenge, with early methods relying on complex temporal modeling like memory queues, space-time graphs, and state-space layers. While these approaches achieved some success, their high computational cost and limited scalability hinder widespread adoption. Recent advances in short video understanding have benefited from deep learning models, but extending these to hours-long content remains difficult due to the need for sophisticated reasoning over extended temporal spans. The emergence of large language models (LLMs) like GPT-4 has opened new avenues, demonstrating impressive capabilities in long-horizon reasoning, document understanding, and decision-making. However, integrating LLMs into long video understanding pipelines requires effective representations of visual content and strategies to handle noisy, redundant information. Existing datasets such as ActivityNet-QA, MovieQA, and EgoSchema reveal limitations, including biases and insufficient modeling of long-range dependencies, motivating the development of more robust, scalable solutions.

The core challenge in long video question-answering (LVQA) is to efficiently and accurately model extended temporal dependencies without resorting to overly complex or resource-intensive architectures. Traditional models rely on elaborate mechanisms like memory modules or graph-based structures, which are costly and difficult to train. Moreover, these models often struggle with noisy or incomplete visual data, leading to degraded performance. The key bottleneck is how to leverage powerful language models' reasoning abilities while effectively representing visual content. Addressing this requires a method that simplifies long-range modeling, filters irrelevant information, and maintains high accuracy in a zero-shot setting, making long video understanding feasible for real-world applications.

This work introduces a two-stage, model-agnostic framework that combines short-term visual captioning with large model reasoning. The first innovation is using pretrained captioners (e.g., LaViLa, BLIP2) to generate textual summaries of short video clips, transforming visual data into language. The second innovation is a multi-round summarization prompt that guides large models to filter noise and focus on relevant content, significantly improving QA accuracy. Unlike prior methods relying on complex temporal modules, this approach leverages the reasoning power of foundation models, making the system simple, flexible, and scalable. The framework supports various captioners and large models, enabling easy adaptation and future improvements.

• �� Segment long videos into short clips (e.g., 1s each).• Use pretrained visual captioners (LaViLa, BLIP2) to generate textual descriptions for each clip.• Concatenate these descriptions in temporal order, forming a comprehensive textual representation.• Design a multi-round prompt: first, ask the model to summarize the noisy captions; second, use this summary to answer specific questions.• Experiment with different captioning models, sampling strategies, and prompt designs to optimize performance.• Conduct ablation studies on component choices, clip length, and sampling frequency.• Evaluate on datasets like EgoSchema, NExT-QA, and IntentQA, focusing on zero-shot accuracy.

The experiments involve evaluating the proposed framework on multiple datasets, including EgoSchema, NExT-QA, and IntentQA, using accuracy as the primary metric. The authors compare different visual captioners (LaViLa, BLIP2, EgoVLP) and large models (GPT-3.5, GPT-4), analyzing the impact of clip length, sampling rate, and prompt design. Ablation studies confirm the effectiveness of multi-round summarization prompts, with LaViLa and GPT-4 achieving the best results. The sampling strategy of 1s clips at 1Hz provides optimal balance between efficiency and accuracy. The system's robustness is validated across datasets, demonstrating its general applicability and zero-shot capabilities.

LLoVi achieves 50.3% accuracy on EgoSchema, outperforming previous methods by 18.1%. LaViLa as captioner yields 55.2%, while GPT-4 reaches 61.2% accuracy. Short clips (1s) sampling outperforms longer segments, with only 2% accuracy drop at 8x efficiency. Multi-round summarization prompts improve accuracy by 3.6%, confirming their importance. The framework's flexibility allows effective adaptation across datasets, establishing a new baseline for zero-shot LVQA. These results highlight the potential of combining pretrained captioners with large models for scalable long video understanding.

This approach can be immediately applied to long video retrieval, automated content moderation, and assistive AI systems in surveillance and entertainment. Its zero-shot nature allows deployment without extensive retraining, suitable for real-world scenarios with diverse content. Long-term, it paves the way for multimodal, real-time video analysis in industries like security, media, and autonomous systems, enabling scalable, intelligent understanding of hours-long videos with minimal supervision.

The system's performance depends heavily on caption quality; noisy or incomplete captions impair reasoning. Sampling strategies may omit critical information in very long videos. Large models like GPT-4 are computationally expensive, limiting practical deployment. Future work should focus on improving caption robustness, reducing inference costs, and integrating multimodal cues to handle complex scenes more effectively.