TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs
TimeLens2 leverages multimodal LLMs with Wasserstein-based rewards for multi-interval video grounding, outperforming models up to 397B parameters.
Key Findings
Methodology
TimeLens2 employs a multi-stage supervision framework, including caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. It constructs reliable multi-interval supervision from 93K long videos, leveraging caption content and independent relocalization. During training, a novel temporal Wasserstein reward computes the exact W1 distance between merged interval supports, providing dense, matching-free feedback for variable-cardinality and fragmented predictions. The model, based on Qwen3-VL, is fine-tuned with reinforcement learning to optimize set-valued interval outputs, significantly improving multi-interval localization across diverse benchmarks.
Key Results
- Across seven benchmarks, TimeLens2-2B achieves an average mIoU of 44.5, surpassing size-matched baselines by 14.2 points. The 4B and 8B variants reach 47.7 and 48.0, outperforming models with up to 397B parameters. The model demonstrates robust performance in datasets like Charades, ActivityNet, and VUE, excelling in long-video, multi-span, and question-based grounding tasks.
- The introduction of Wasserstein rewards effectively addresses fragmentation and unequal interval cardinalities, providing continuous geometric guidance. Multi-stage supervision ensures reliable long-video evidence annotation, aligning training with evaluation metrics. The model's architecture and training strategies lead to state-of-the-art results, validating the approach's generality.
- The combination of geometric-aware rewards and staged supervision enables the model to handle complex, multi-region, and multi-length scenarios, marking a significant advancement in multimodal video understanding.
Significance
This work addresses longstanding challenges in long-video multi-region temporal grounding, notably supervision reliability, matching issues, and fragmentation. The Wasserstein-based reward introduces a continuous, geometry-sensitive metric that overcomes the limitations of traditional IoU. The model's high performance across diverse benchmarks demonstrates its potential to revolutionize video content retrieval, event detection, and multimedia analysis. It paves the way for more reliable, scalable, and generalizable video understanding systems, impacting both academia and industry by enabling more precise and explainable video localization and retrieval.
Technical Contribution
The paper introduces a novel set-based temporal grounding framework that models evidence as interval sets throughout supervision and optimization. It innovates with a Wasserstein distance-based reward that provides dense, partition-invariant feedback, overcoming issues with fragmentation and unequal interval cardinalities. The multi-stage supervision pipeline, combining caption-based proposals, independent relocalization, consensus, and boundary refinement, ensures high-quality labels for long videos. The model architecture, training regimen, and reward design collectively push the state-of-the-art in multi-interval video grounding, offering theoretical guarantees of geometric consistency and practical improvements in diverse scenarios.
Novelty
This research pioneers the explicit modeling of multi-region evidence as variable-cardinality interval sets within a unified framework, integrating a Wasserstein distance-based reward for set-level optimization. Unlike prior methods relying on IoU or pairwise matching, it introduces a continuous, fragmentation-invariant geometric measure, enabling robust training and inference in complex, multi-interval scenarios. The staged supervision pipeline further enhances label reliability, setting a new standard for long-video multi-region grounding.
Limitations
- Despite improvements, the model may struggle with extremely long videos or highly sparse evidence, where boundaries become ambiguous. The reliance on high-quality captions and multi-stage supervision increases annotation complexity and cost.
- Real-time inference remains challenging due to computational demands of the multi-stage pipeline and Wasserstein calculations. Further optimization is needed for deployment in latency-sensitive applications.
- Generalization to unseen domains or highly dynamic scenes requires additional adaptation, as current training heavily depends on web-sourced datasets with rich annotations.
Future Work
Future directions include optimizing the inference speed, exploring self-supervised pretraining to reduce annotation costs, and extending the framework to multi-task learning such as event detection and action recognition. Incorporating more efficient geometric metrics and scalable training strategies will further enhance real-world applicability. Additionally, expanding datasets to cover more diverse scenarios and developing domain adaptation techniques will improve model robustness and generalization.
AI Executive Summary
In an era where video archives grow exponentially, the challenge of accurately locating specific moments within long videos remains a critical hurdle. Traditional methods, often limited to short clips or single-event annotations, fall short when faced with complex, multi-region evidence scattered across hours of footage. Addressing this gap, TimeLens2 introduces a groundbreaking approach that combines multimodal large language models with a geometric-aware reinforcement learning framework.
The core innovation lies in modeling evidence as a set of variable-length intervals, enabling the system to handle multiple disjointed moments simultaneously. To achieve this, the authors develop a multi-stage supervision pipeline, starting with caption-derived proposals that generate coarse candidate regions. These are then independently relocalized, verified through semantic and temporal consensus, and refined at the boundaries, ensuring high-quality labels even in noisy long-video data.
Complementing this, the paper proposes a novel temporal Wasserstein reward, which measures the exact transportation cost between predicted and ground-truth interval supports. Unlike traditional IoU-based metrics, this continuous, partition-invariant measure effectively guides the model to improve in fragmented and unequal-cardinality scenarios, addressing a longstanding challenge in multi-interval localization.
Extensive experiments across seven benchmarks—including Charades, ActivityNet, and VUE—demonstrate that TimeLens2 achieves state-of-the-art performance. The models with 2B, 4B, and 8B parameters outperform existing solutions, with the largest variant surpassing models with 397B parameters. These results highlight the method’s robustness, scalability, and broad applicability in diverse video understanding tasks.
This work not only advances theoretical understanding by integrating set-based geometry into reinforcement learning but also offers practical tools for video retrieval, event detection, and multimedia analysis. Despite some limitations in real-time deployment and extreme scenarios, the approach sets a new standard for reliable, scalable, and interpretable long-video localization, promising significant impact in both academia and industry.
Deep Dive
Plain Language Accessible to non-experts
想象你在整理一堆散乱的录像片段，试图找到所有关于某个事件的关键瞬间，比如一场派对。传统的方法就像用放大镜逐个检查每个片段，费时又容易错过重要内容。而TimeLens2像是有一台聪明的机器人助手，它可以自动扫描这些视频，识别出所有可能的关键片段，并用一种特别的“距离”算法，判断哪些片段最接近你想找的证据。它会不断确认这些片段是否确实是你要找的内容，然后把它们的边界精准划出来。这样一来，无论视频多长、多散乱，它都能帮你快速找到所有关键时刻，节省大量时间。这就像有个超级厉害的助手，帮你整理和定位视频中的重要瞬间，让你不用逐个翻查，轻松找到想要的内容。
ELI14 Explained like you're 14
想象你在看一堆视频，比如学校的录像或者朋友的聚会照片。你想找到所有关于某个事件的片段，比如有人吃蛋糕或者玩游戏。以前的方法就像用放大镜一块块看，既慢又容易错过重点。现在，TimeLens2就像有个超级聪明的机器人助手，它可以自动扫描这些视频，找到所有相关的片段。它用一种特别的数学方法，像在地图上画线，把相关的片段连接起来，不管这些片段是不是连续的。它还会不断确认这些片段是不是你要找的内容，确保没有误会。最后，它能把这些片段的边界划得很精准，让你一眼就能看到所有重要的瞬间。这样一来，找视频里的关键时刻变得既快又准，就像有个超级厉害的助手帮你整理视频，省时又省力！
Abstract
Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional \(W_1\) between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.