VideoExplorer: Think With Videos For Agentic Long-Video Understanding

Long-video understanding has evolved from early frame-based detection to recent multimodal models like Florence and Gato, aiming to handle extended sequences. Despite progress, existing methods struggle with information overload, static representations, and limited reasoning depth. Static downsampling sacrifices details, while static textual summaries lose visual nuances. Recent retrieval-augmented approaches attempt to mitigate these issues but remain constrained by task-agnostic preprocessing and limited dynamic perception. The demand for models capable of multi-step, task-oriented, and interpretable reasoning over hours-long videos remains unmet, especially in real-world applications requiring high fidelity and scalability.

Current long-video understanding methods face bottlenecks such as information loss due to frame downsampling, static representations that cannot adapt to task demands, and limited reasoning depth. These issues lead to reduced accuracy, poor interpretability, and inability to handle complex multi-hop scenarios. The core challenge is to develop a system that can dynamically focus on relevant video segments, perform multi-step reasoning, and produce faithful, interpretable answers without excessive computational costs. Addressing these bottlenecks is critical for deploying AI in real-world, safety-critical applications.

The paper introduces the 'thinking with video' paradigm, emphasizing dynamic, task-aware exploration. Key innovations include:

1) Planning module: decomposes complex tasks into sub-questions, guiding the reasoning process.

2) Temporal grounder: localizes relevant video segments, ensuring evidence relevance.

3) Scalable perception: adjusts sampling granularity dynamically, balancing detail and efficiency.

4) Dataset construction: uses difficulty-adaptive sampling to generate challenging reasoning trajectories.

5) Two-stage training: supervised imitation for initial learning, followed by trajectory preference optimization for refinement.

These innovations collectively enable multi-step, interpretable, and scalable long-video reasoning, surpassing static or single-pass methods.

• �� The planner analyzes the task, generating sub-questions and reasoning paths.
• �� The temporal grounder maps sub-questions to specific video segments, verifying relevance via visual-language models.
• �� The perception module dynamically adjusts frame sampling density based on task complexity—dense sampling for fine-grained queries, coarse sampling for global questions.
• �� A high-quality reasoning dataset is built using difficulty-adaptive sampling, emphasizing complex cases.
• �� Training involves supervised fine-tuning (SFT) to imitate expert trajectories, then trajectory-level preference optimization (TDPO) to refine reasoning paths.
• �� During inference, the model iteratively generates sub-questions, localizes relevant segments, and integrates information to produce answers.

The evaluation uses LVBench, MLVU, and other long-video datasets, comparing VideoExplorer with baselines like static reasoning models and retrieval-augmented methods. Metrics include accuracy and [email protected] for temporal grounding. Ablation studies assess the impact of each component—temporal grounder, preference optimization, dataset difficulty. Hyperparameters include 7B and 32B model scales, training with Adam optimizer, learning rate 1e-5, batch size 32. Results show significant performance gains, especially in multi-hop reasoning and fine-grained localization, confirming the effectiveness of the framework across diverse scenarios.

VideoExplorer achieves 54.3% average accuracy on LVBench and MLVU, outperforming static models by over 10%. Temporal [email protected] reaches 27.8%, a substantial improvement over baseline models (~16-20%). Ablation results reveal that removing the temporal grounder or preference optimization reduces accuracy by 5-8%, highlighting their importance. The model demonstrates robustness in complex reasoning tasks, effectively localizing key moments and integrating multimodal cues, confirming its scalability and interpretability. These results validate the effectiveness of the dynamic, task-aware approach for long-video understanding.

The framework can be applied to autonomous driving for scene understanding, security surveillance for event detection, and robotic systems for behavior analysis. Its ability to perform multi-step, interpretable reasoning over hours-long videos makes it suitable for safety-critical applications requiring high accuracy and explainability. Future integration with real-time processing and multi-modal signals like audio and text will expand its deployment in industry, enabling smarter, more autonomous systems.

Despite its advantages, the model faces challenges in ultra-long videos with highly ambiguous or noisy content, which can impair localization accuracy. The training process requires extensive high-quality reasoning data, increasing annotation costs. Real-time inference remains computationally demanding, especially at higher resolutions. Future work should focus on optimizing speed, reducing resource consumption, and enhancing robustness in diverse, real-world scenarios.