InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
InternVid leverages large-scale video-text data and multi-scale captioning to train ViCLIP, achieving state-of-the-art zero-shot action recognition.
Key Findings
Methodology
This work introduces InternVid, a massive dataset with over 7 million videos and 234 billion words, generated via multi-scale descriptions using large language models (LLMs). The dataset employs coarse and fine captioning strategies—middle frame captioning with BLIP2 and frame-by-frame summarization with Tag2Text—to enhance semantic richness. A contrastive learning framework based on CLIP architecture, with a ViT-L backbone, is used to train ViCLIP. Video masking accelerates training, and the model is validated on multiple benchmarks, demonstrating superior zero-shot action recognition and retrieval capabilities.
Key Results
- ViCLIP trained on InternVid achieves 75.7%, 73.5%, and 66.4% top-1 accuracy on Kinetics 400/600/700 datasets in zero-shot settings, surpassing previous models by over 5%.
- On video retrieval tasks, ViCLIP outperforms baselines on MSR-VTT and YouCook2, with significant improvements in retrieval precision.
- In multimodal dialogue and video generation tasks, ViCLIP demonstrates strong transferability, producing coherent, contextually relevant outputs, validating the dataset's quality.
Significance
This research addresses the bottleneck of limited high-quality video-text data, enabling scalable pretraining of multimodal models. The large-scale dataset and the ViCLIP model significantly advance the state of the art, facilitating applications in video understanding, content creation, and human-computer interaction. The automated data generation approach reduces annotation costs and accelerates progress in unsupervised learning, with broad implications for academia and industry.
Technical Contribution
The paper introduces a multi-scale captioning pipeline powered by LLMs to generate high-quality, diverse video descriptions. It combines video masking with contrastive learning in a ViT-L framework, creating a scalable, efficient training paradigm. The dataset's unprecedented scale enables training models that outperform existing approaches in zero-shot scenarios. The integration of multi-scale descriptions and masking strategies represents a novel contribution to large-scale multimodal pretraining.
Novelty
This is the first work to systematically construct a 7-million-video, 234-billion-word dataset with multi-scale descriptions generated automatically via LLMs. The combination of multi-scale captioning, video masking, and contrastive learning in a unified framework sets a new benchmark for large-scale video-text pretraining, surpassing prior efforts like WebVid and HowTo100M.
Limitations
- Despite its scale, the dataset is predominantly sourced from YouTube, which may introduce domain bias and limit generalization to other content types or languages.
- Generated descriptions, while diverse, may lack the fine-grained detail necessary for certain applications like detailed action understanding.
- Training large models like ViCLIP requires substantial computational resources, limiting accessibility for smaller research groups.
Future Work
Future directions include expanding multilingual and multi-scene datasets, integrating audio and other modalities for richer representations, and optimizing model architectures for efficiency. Further research will explore fine-grained action understanding, domain adaptation, and real-time inference capabilities to broaden practical deployment.
AI Executive Summary
The exponential growth of online video content has created a pressing need for advanced multimodal understanding systems capable of interpreting complex scenes and actions. Traditional datasets, such as MSR-VTT and YouCook2, are limited in scale and semantic richness, constraining the development of robust models. To address this, the authors introduce InternVid, a massive video-text dataset comprising over 7 million videos and 234 billion words, generated automatically through a multi-scale captioning pipeline powered by large language models (LLMs). This approach ensures high semantic fidelity and diversity, covering 16 scenarios and thousands of actions across multiple languages.
Building on this dataset, the researchers develop ViCLIP, a contrastive learning model based on ViT-L, trained to align video and textual representations. The model incorporates a novel video masking strategy, inspired by masked autoencoders, to accelerate training and improve robustness. Extensive experiments demonstrate ViCLIP’s superior performance in zero-shot action recognition, achieving over 75% accuracy on Kinetics 400/600/700, surpassing previous benchmarks. Its effectiveness extends to video retrieval and multimodal understanding tasks, validating the dataset’s quality and the training methodology.
This work significantly advances the field by enabling scalable, unsupervised pretraining of multimodal models. The large-scale, automatically generated dataset reduces reliance on manual annotations, fostering rapid progress in video understanding, content creation, and human-computer interaction. Future research will focus on expanding multilingual data, integrating additional modalities, and optimizing models for real-world deployment. Despite current limitations, such as domain bias and computational costs, the proposed framework paves the way for next-generation multimodal AI systems, opening new horizons for both academia and industry.
Deep Dive
Abstract
This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.