Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoning
Proposes Video-in-the-Loop (ViTL), a two-stage long-video QA framework combining localization and span-aware answering, achieving up to 8.6% improvement with 50% less input.
Key Findings
Methodology
ViTL employs a two-stage process: first, a low-fps skim scans the entire video to localize relevant segments using IoU-based optimization; second, the model zooms into these segments at a higher frame rate, reallocating visual tokens to focus on evidence. Training utilizes Group-Relative Policy Optimization (GRPO), which couples temporal IoU with answer correctness, enabling end-to-end learning that aligns span localization with answer utility. The dataset VGrounding-QA converts event graphs into span-grounded multiple-choice questions, providing structured supervision. This approach allows the model to jointly optimize localization and answering, improving accuracy and interpretability under fixed token budgets.
Key Results
- On Charades-STA and ActivityNet-Captions, ViTL achieves up to 8.6% accuracy gains, with 50% fewer frames, outperforming uniform sampling baselines. Recall@IoU=0.5 reaches 63.5%, surpassing previous models. Ablation studies show span-aware token reallocation consistently outperforms uniform sampling, with an average 4.2% boost in localization and QA performance. The end-to-end training effectively couples span localization with answer correctness, demonstrating robustness across datasets.
- In long-video QA and temporal grounding tasks, ViTL demonstrates superior efficiency and accuracy, maintaining high performance with limited computational resources. The integration of event knowledge graphs enhances the model’s understanding of complex multi-span events, leading to better reasoning over extended temporal contexts. The experimental results validate the effectiveness of the two-stage approach and the structured supervision scheme.
- Overall, ViTL advances long-video understanding by enabling precise, interpretable, and compute-efficient localization and answering. It addresses the challenge of information sparsity and background dilution, providing a scalable solution suitable for real-world applications like video retrieval, content analysis, and automated surveillance.
Significance
This work addresses fundamental challenges in long-video comprehension—efficiently locating relevant evidence and generating accurate answers—by integrating structured event knowledge and a novel two-stage framework. It significantly reduces computational costs while improving interpretability, making large-scale video analysis feasible. The coupling of span localization with answer correctness via end-to-end training marks a step forward in bridging the gap between localization and reasoning tasks. The approach’s scalability and explainability open new avenues for deploying AI in real-world scenarios such as video content moderation, intelligent surveillance, and multimedia retrieval, impacting both academia and industry.
Technical Contribution
The paper introduces a novel two-stage architecture that reallocates visual tokens based on predicted evidence, coupled with a group-relative policy optimization strategy that jointly trains span localization and answer generation. The innovative use of event knowledge graphs for constructing span-grounded QA datasets provides rich supervision signals, enabling the model to learn multi-span reasoning. The end-to-end training paradigm ensures that span quality directly influences answer accuracy, a significant departure from traditional isolated localization and QA models. These contributions collectively push the boundary of scalable, interpretable long-video understanding.
Novelty
This research is the first to combine event-graph structured supervision with a two-stage, span-aware token reallocation framework for long-video question answering. Unlike prior methods relying solely on uniform sampling or heuristic key-frame selection, ViTL dynamically focuses computational resources on evidence-rich segments, guided by structured supervision. The integration of end-to-end training with a group-relative policy for joint optimization of localization and answering sets a new standard, enabling scalable and interpretable long-video analysis.
Limitations
- Despite improvements, the model still faces challenges in accurately localizing highly complex or multi-event scenes with rapid scene changes, which can affect span accuracy.
- Training relies heavily on manually annotated event graphs, which are costly to produce and may limit generalization to unseen or unstructured videos.
- Computational costs, especially for very long videos, remain significant; further optimization is needed for real-time applications.
Future Work
Future directions include integrating multi-modal cues such as audio and text for richer reasoning, automating event graph construction via self-supervised methods, and optimizing model architectures for real-time deployment. Expanding the dataset to cover more diverse scenarios and reducing annotation costs will further enhance generalization. Additionally, exploring unsupervised or weakly supervised learning paradigms could make the approach more scalable to large-scale, unstructured video repositories.
AI Executive Summary
Understanding long videos remains a significant challenge in artificial intelligence, primarily due to the vast amount of information and the sparse distribution of relevant evidence. Traditional approaches often rely on uniform sampling, which tends to waste computational resources on background content, making it difficult to accurately locate and reason about key events. This limitation hampers the development of scalable, interpretable, and efficient long-video question answering systems.
To address these issues, this paper introduces Video-in-the-Loop (ViTL), a novel two-stage framework that intelligently allocates computational effort. In the first stage, a low-fps skim scans the entire video, localizing potential evidence segments using an IoU-based optimization. This step ensures that the model focuses on promising regions, reducing background noise. In the second stage, the model zooms into these segments at a higher frame rate, reallocating visual tokens to evidence-rich areas, and performs detailed reasoning to generate multiple-choice answers. This span-aware reallocation significantly improves both localization accuracy and answer correctness.
A key innovation is the training strategy employing Group-Relative Policy Optimization (GRPO), which couples the localization quality with answer utility, enabling the model to learn how to prioritize evidence effectively. Additionally, the authors construct VGrounding-QA, a dataset that converts event graphs into span-grounded multiple-choice questions, providing rich supervision signals for training. This structured approach allows the model to learn multi-span reasoning and improves interpretability.
Experimental results on datasets such as Charades-STA, ActivityNet-Captions, and LongVideoBench demonstrate that ViTL outperforms existing methods, achieving up to 8.6% accuracy improvements while using 50% fewer frames. Ablation studies confirm that span-aware token reallocation consistently surpasses uniform sampling, validating the effectiveness of the approach.
Overall, ViTL offers a scalable, interpretable, and compute-efficient solution for long-video question answering, with broad implications for multimedia analysis, content retrieval, and AI understanding of extended temporal contexts. Future work will focus on multi-modal integration, reducing annotation costs, and enhancing real-time capabilities, paving the way for more intelligent and explainable video understanding systems.
Deep Dive
Glossary
IoU (Intersection over Union)
A metric measuring the overlap between predicted and ground-truth segments, used to evaluate localization accuracy. In this paper, IoU guides the training of segment localization.
Used in the first stage to optimize segment proposals.
Group-Relative Policy Optimization (GRPO)
A reinforcement learning strategy that optimizes multiple responses by comparing their rewards relative to each other, promoting better policy learning. Applied here to jointly train span localization and answer generation.
Core training mechanism coupling localization and QA.
VGrounding-QA
A dataset construction method converting event graphs into span-grounded multiple-choice questions, providing structured supervision for temporal localization and reasoning.
Enables supervised training of span-aware models.
Event Knowledge Graph
A structured representation of events, objects, and their relationships within videos, used to generate complex reasoning questions.
Supports multi-hop reasoning and supervision.
Span-grounded QA
Question answering where the relevant temporal segments (spans) are explicitly annotated and used to guide the model’s focus.
Key to interpretability and precise localization.
Open Questions Unanswered questions from this research
- 1 如何进一步减少对人工事件图谱的依赖，提升模型在未标注视频中的泛化能力？
- 2 模型在极端复杂或多事件场景中的表现仍有限，如何提升多跨度事件的准确定位？
- 3 未来是否可以结合自监督学习，减少对标注数据的依赖，提升大规模应用的可行性？
Applications
Immediate Applications
视频内容检索
利用模型快速定位视频中的关键信息，提升内容检索的效率和准确性，适用于视频平台和监控系统。
智能监控与安全
在监控视频中自动检测和定位异常事件，辅助安全人员快速响应，提高监控系统的智能化水平。
Long-term Vision
自动视频摘要与理解
结合长视频理解技术，生成简洁、准确的内容摘要，推动智能内容管理和虚拟助手的发展。
Abstract
We present \emph{Video-in-the-Loop} (ViTL), a two-stage long-video QA framework that preserves a fixed token budget by first \emph{localizing} question-relevant interval(s) with a low-fps skim and then \emph{answering} via span-aware reallocation of visual tokens at higher effective frame rate, emitting an interleaved output with both spans and the final option for direct attribution. We also introduce \dataname{}, which converts description based event graphs into \emph{span-grounded} multiple-choice QA by pairing each question with \emph{ground-truth} time span(s) and related reasoning. ViTL is trained end-to-end with an interleaved group-relative objective that couples temporal IoU for localization with answer correctness, allowing credit to flow from answers back to spans without increasing compute. Under fixed token budgets, ViTL attains up to 8.6% with 50% less frame input on long-video QA and temporal grounding (e.g., Charades-STA, ActivityNet-Captions) and ablations show that span-aware token reallocation consistently surpasses uniform sampling. Together, \dataname{} and ViTL provide an interpretable, compute-efficient recipe for scalable long-video QA.