TALL: Temporal Activity Localization via Language Query
Proposes CTRL, a cross-modal temporal regression model for language-based action localization, achieving significant improvements in IoU=0.5 (45.2% R@1) over state-of-the-art.
Key Findings
Methodology
This paper introduces a Cross-modal Temporal Regression Localizer (CTRL) framework that jointly models video and language features for precise temporal action localization. Visual features are extracted via a 3D CNN (C3D), capturing spatiotemporal cues, while sentence embeddings are obtained through LSTM or Skip-thought encoders. The multi-modal fusion module combines these features through element-wise addition, multiplication, and concatenation, creating a joint representation. The core of CTRL involves a multi-task learning setup with alignment scoring and boundary regression. Boundary offsets are predicted using a non-parameterized approach (start/end offsets) and a parameterized approach (center and length offsets). The training employs a multi-scale sliding window sampling strategy, with a combined loss function that balances alignment and regression objectives. The model is trained on datasets including TaCoS and a newly constructed Charades-STA, which annotates sentence-level temporal boundaries. Experimental evaluation uses recall at different IoU thresholds (R@n, IoU=m), demonstrating that CTRL outperforms existing methods, especially at IoU=0.5, with a R@1 of 45.2%.
Key Results
- CTRL achieves R@1=45.2% at IoU=0.5 on TaCoS, surpassing previous methods by over 12 percentage points, demonstrating superior boundary precision and matching capability.
- On Charades-STA, CTRL reaches R@5=68.4% and IoU=0.3=72.1%, outperforming baseline sliding window and classification approaches, validating its robustness and generalization.
- Ablation studies confirm that non-parameterized boundary regression outperforms parameterized schemes, and Skip-thought embeddings outperform word2vec+LSTM, especially for complex and long sentences.
Significance
This work addresses the limitations of traditional action detection methods confined to fixed categories, enabling flexible natural language queries for temporal localization. It significantly advances video understanding by allowing open-set, user-defined activity searches, which is crucial for applications like intelligent surveillance, video retrieval, and human-computer interaction. The proposed framework bridges the gap between visual content and linguistic descriptions, facilitating more natural and intuitive interactions with video data. Moreover, the introduction of the Charades-STA dataset enriches research resources, fostering further innovations in cross-modal video analysis.
Technical Contribution
The paper introduces a novel CTRL framework that integrates visual and textual features through multi-modal fusion, employing a multi-task training strategy that combines alignment scoring with boundary regression. The boundary regression employs a non-parameterized offset scheme, which proves more effective than traditional parameterized offsets in complex scenarios. The architecture leverages multi-scale sliding window sampling to generate training samples, enhancing the model's robustness. The experimental results demonstrate that CTRL achieves state-of-the-art performance on challenging datasets, validating its effectiveness and paving the way for future research in open-vocabulary, language-driven video localization.
Novelty
This is the first work to formulate the problem of natural language-based temporal activity localization (TALL). It innovatively combines cross-modal feature fusion with boundary regression, moving beyond traditional classification-based detection approaches. The non-parameterized boundary offset scheme and the joint modeling of alignment and regression tasks constitute key breakthroughs, enabling precise localization of complex activities described in natural language. The creation of Charades-STA with sentence-level temporal annotations further supports the research community, marking a significant step forward in open-set, language-guided video understanding.
Limitations
- The model's performance degrades with highly complex or ambiguous sentences, especially those with multiple clauses or vague descriptions, indicating a need for more sophisticated language understanding modules.
- Training relies on extensive annotated datasets; automatic annotation methods may introduce noise, limiting generalization to unseen or real-world scenarios.
- Boundary regression, although improved with non-parameterized offsets, can still be affected by background clutter and motion blur, especially in low-quality videos. Real-time deployment remains challenging due to computational costs.
Future Work
Future research will focus on integrating attention mechanisms to better capture relevant temporal segments and contextual cues. Expanding the model to handle multi-sentence and multi-activity queries, as well as multi-lingual descriptions, will broaden its applicability. Additionally, exploring unsupervised or weakly supervised learning paradigms could reduce reliance on extensive annotations. Real-time inference and deployment in practical systems like surveillance or assistive robots are also promising directions.
AI Executive Summary
In an era where video content proliferates exponentially across social media, surveillance, and entertainment platforms, the challenge of efficiently and accurately locating specific actions within untrimmed videos becomes increasingly critical. Traditional approaches predominantly rely on pre-defined action categories and sliding window detection, which, while effective in controlled settings, falter when faced with the complexity and diversity of real-world scenarios. These methods often lack the flexibility to interpret natural language queries, limiting their utility in user-centric applications.
Addressing this gap, the authors introduce a groundbreaking framework called Cross-modal Temporal Regression Localizer (CTRL), designed to enable natural language-driven temporal action localization. The core idea is to allow users to describe an activity in plain language, such as βa person runs to the window and looks out,β and have the system accurately identify the corresponding time segment within a long, untrimmed video. This approach significantly enhances the flexibility and usability of video retrieval systems, making them more accessible to non-expert users.
The CTRL framework integrates advanced deep learning components, including a 3D convolutional neural network (C3D) for visual feature extraction, capturing spatiotemporal cues from video clips. Simultaneously, it employs sentence encoders like LSTM or Skip-thought to embed natural language descriptions into a shared semantic space. The multi-modal fusion module combines these features through element-wise addition, multiplication, and concatenation, enabling rich cross-modal interactions. The model then predicts alignment scores indicating how well a clip matches the query, alongside boundary offsets to precisely locate the start and end times of the activity.
A key innovation lies in the boundary regression scheme. Unlike traditional parameterized offsets that predict relative shifts, the authors propose a non-parameterized approach directly estimating start and end point offsets. This method proves more effective in complex scenes with background noise or ambiguous actions. The training employs a multi-task loss function balancing alignment and regression objectives, utilizing multi-scale sliding window sampling to generate diverse training samples. The datasets used include TaCoS and a newly constructed Charades-STA, which annotates sentence-level temporal boundaries, along with complex sentence queries for rigorous testing.
Experimental results demonstrate that CTRL surpasses existing methods significantly. On the TaCoS dataset, it achieves a R@1 of 45.2% at IoU=0.5, outperforming previous models by over 12 percentage points. Similarly, on Charades-STA, it reaches R@5=68.4% and IoU=0.3=72.1%, validating its robustness and generalization. Ablation studies confirm the superiority of non-parameterized boundary regression and the effectiveness of Skip-thought embeddings, especially for complex sentences and longer videos.
This research marks a substantial advancement in video understanding, bridging the gap between visual content and natural language. It opens new avenues for user-friendly video retrieval, intelligent surveillance, and human-computer interaction, where natural language queries can be reliably translated into precise temporal locations. Future work will focus on incorporating attention mechanisms, expanding to multi-sentence queries, and optimizing for real-time deployment, aiming to make intelligent video analysis more intuitive, flexible, and scalable.
Deep Dive
Abstract
This paper focuses on temporal localization of actions in untrimmed videos. Existing methods typically train classifiers for a pre-defined list of actions and apply them in a sliding window fashion. However, activities in the wild consist of a wide combination of actors, actions and objects; it is difficult to design a proper activity list that meets users' needs. We propose to localize activities by natural language queries. Temporal Activity Localization via Language (TALL) is challenging as it requires: (1) suitable design of text and video representations to allow cross-modal matching of actions and language queries; (2) ability to locate actions accurately given features from sliding windows of limited granularity. We propose a novel Cross-modal Temporal Regression Localizer (CTRL) to jointly model text query and video clips, output alignment scores and action boundary regression results for candidate clips. For evaluation, we adopt TaCoS dataset, and build a new dataset for this task on top of Charades by adding sentence temporal annotations, called Charades-STA. We also build complex sentence queries in Charades-STA for test. Experimental results show that CTRL outperforms previous methods significantly on both datasets.
References (20)
Deep Visual-Semantic Alignments for Generating Image Descriptions
A. Karpathy, Li Fei-Fei
Learning Spatiotemporal Features with 3D Convolutional Networks
Du Tran, Lubomir D. Bourdev, R. Fergus et al.
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Gunnar A. Sigurdsson, GΓΌl Varol, X. Wang et al.
Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
Zheng Shou, Dongang Wang, Shih-Fu Chang
Large-Scale Video Classification with Convolutional Neural Networks
A. Karpathy, G. Toderici, Sanketh Shetty et al.
Two-Stream Convolutional Networks for Action Recognition in Videos
K. Simonyan, Andrew Zisserman
Skip-Thought Vectors
Ryan Kiros, Yukun Zhu, R. Salakhutdinov et al.
The Stanford CoreNLP Natural Language Processing Toolkit
Christopher D. Manning, M. Surdeanu, John Bauer et al.
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Junhua Mao, Wei Xu, Yi Yang et al.
Long-term recurrent convolutional networks for visual recognition and description
Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach et al.
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Sahar Kazemzadeh, Vicente Ordonez, M. Matten et al.
Very Deep Convolutional Networks for Large-Scale Image Recognition
K. Simonyan, Andrew Zisserman
Beyond short snippets: Deep networks for video classification
Joe Yue-Hei Ng, Matthew J. Hausknecht, Sudheendra Vijayanarasimhan et al.
Visual Semantic Search: Retrieving Videos via Complex Textual Queries
Dahua Lin, S. Fidler, Chen Kong et al.
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Ross B. Girshick, Jeff Donahue, Trevor Darrell et al.
Distributed Representations of Words and Phrases and their Compositionality
Tomas Mikolov, I. Sutskever, Kai Chen et al.
Grounding Action Descriptions in Videos
Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel et al.
Script Data for Attribute-Based Recognition of Composite Activities
Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka et al.
A database for fine grained activity detection of cooking activities
Marcus Rohrbach, S. Amin, Mykhaylo Andriluka et al.
Action recognition by dense trajectories
Heng Wang, Alexander Klaser, Cordelia Schmid et al.
Cited By (20)
The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding
Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval
TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs
Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs
DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding
Your VLM Already Knows When: Training-Free Temporal Grounding by Asking Yes or No
Video-Text Temporal Localization via Multi-Scale Convolution and Dynamic Routing
Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding
EventCoT: Event-centric Video Chain-of-thought for Reasoning Temporal Localization
LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
Generation-Augmented Video Corpus Moment Retrieval
A Multi-Granularity Game-Theoretic Approach to Weakly Supervised Temporal Article Grounding
AI-Based semantic video indexing: a unified survey from segmentation to retrieval
OmniReasoner: Thinking with Long Audio-Video via Native Tool Use
A Reciprocal Interaction Framework for Collaborative Temporal Grounding and Question Answering in Egocentric Videos
Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
TimePLE: Rethinking Temporal Representation for Video Temporal Grounding
An Episode Memory-Guided Dual-Stage Framework for Long-Form Video Temporal Grounding
ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding