Online Video Agent Harness for Long Video Understanding

Long video understanding is a key multimodal application and a practical test of the long-context capabilities of large models. Long-form videos, such as short films, animations, vlogs, and movies, may run from tens of minutes to several hours. Even with sparse sampling at 1 FPS, such videos can contain thousands of frames. Under the standard processing paradigm for large multimodal models, this can easily lead to contexts with millions of tokens, far beyond the effective context length of most models, thus prone to context degradation.

The core problem in long video understanding is that query-relevant evidence is sparsely distributed across long temporal spans, while packing dense frames into a single VLM context incurs context rot and high costs. Existing video agents often rely on query-agnostic offline preprocessing or ad hoc tool sets, which can miss query-specific details and waste computation.

The core innovation of VideoXAgent lies in its purely online video-agent framework, which starts from the given video file and user query, plans and decomposes the task, invokes specialized expert tools on demand, and aggregates multimodal evidence to produce a final answer. It utilizes a data-driven taxonomy of capabilities and heterogeneous expert tools, dynamically invoking scripts, VLMs, and domain models (e.g., detection, OCR, ASR, face recognition) to address context rot and high costs in long video understanding.

• �� Use a data-driven capability taxonomy to guide tool design. • Design 60+ expert tools and compositional workflows, each targeting a specific atomic sub-problem. • At inference time, the system adapts to the query and video duration, decomposes the query into sub-problems, invokes appropriate tools, aggregates multimodal evidence, and controls the execution budget to prevent unbounded loops.

Experiments were conducted on Video-MME-Long, LongVideoBench-Long, LVBench, and MINERVA, demonstrating that VideoXAgent performs comparably to frontier LMMs and video agents under a smaller context footprint. Notably, on complex video-reasoning benchmarks such as MINERVA, it matches this level while using only about 15% of the context of a 1,024-frame dense-packing baseline.

VideoXAgent achieves 85.1% accuracy on Video-MME-Long, 76.5% on LVBench, and 65.7% on MINERVA with about 50k agent context, even on hour-long videos.

VideoXAgent can be applied to various scenarios requiring long video understanding, such as video surveillance, film analysis, and educational video parsing.

Despite its strong performance in long video understanding, VideoXAgent may still face time and resource consumption issues when handling extremely long videos due to tool invocation. Additionally, dependency on tools may lead to performance degradation if tools are imperfect. Future work could explore more efficient tool invocation strategies and broader domain model integration to further enhance the efficiency and accuracy of long video understanding.