InternVideo: General Video Foundation Models via Generative and Discriminative Learning
InternVideo combines generative and discriminative self-supervised learning, achieving 91.1% top-1 accuracy on Kinetics-400 across 39 video tasks.
Key Findings
Methodology
InternVideo employs masked video modeling (VideoMAE) and video-language contrastive learning (based on CLIP), integrated via a learnable cross-modal attention (CMA) module. The architecture uses ViT and UniformerV2 as encoders, trained on 12 million videos from diverse domains. The model combines generative and discriminative signals, enabling multi-task transfer. During pretraining, VideoMAE reconstructs masked spatiotemporal patches, while contrastive learning aligns video and text embeddings. The CMA dynamically fuses features, improving performance across tasks like action recognition, detection, and video-language retrieval.
Key Results
- Achieved 91.1% top-1 accuracy on Kinetics-400, surpassing previous SOTA methods such as MaskFeat and CoCa. On Something-Something V2, reached 77.2%, demonstrating robustness. The model outperforms existing approaches on 39 datasets across action recognition, detection, and video QA, confirming its generalization. Ablation studies show that combining generative and discriminative features yields better results than either alone, with cross-modal attention further boosting performance.
Significance
This work advances the field by providing a unified, scalable video foundation model capable of handling diverse tasks with high accuracy. It addresses the longstanding challenge of creating a general-purpose video understanding system, bridging the gap between task-specific models and broad applicability. The integration of generative and discriminative learning paradigms offers a new pathway for multi-task, multi-modal AI, with significant implications for industry applications such as content retrieval, video editing, and autonomous systems. The model’s strong transferability and efficiency set a new benchmark, inspiring future research in scalable, versatile video AI.
Technical Contribution
The core innovation lies in combining VideoMAE’s masked autoencoding with CLIP-based contrastive learning within a unified framework, enhanced by a learnable cross-modal attention module. This enables dynamic feature fusion, leveraging the strengths of both generative and discriminative signals. The architecture supports large-scale pretraining on massive datasets, with modular design allowing efficient scaling. The approach also introduces a multi-task training pipeline, validated across 39 datasets, demonstrating superior performance in action recognition, detection, and multimodal tasks. This work pushes the boundary of multi-task, multi-modal video foundation models.
Novelty
This is the first work to explicitly unify masked autoencoding and contrastive multimodal learning for comprehensive video understanding. The introduction of a learnable cross-modal attention mechanism for feature fusion distinguishes it from prior models that treat these paradigms separately. Unlike existing models focused on single tasks or modalities, InternVideo achieves broad generalization across diverse video tasks, setting a new paradigm for scalable, multi-task video foundation models.
Limitations
- The model’s large size and computational demands limit deployment on resource-constrained devices. Future work should explore model compression and efficiency improvements. In extremely complex or long-duration videos, performance gaps remain, especially in fine-grained action recognition. Additionally, reliance on massive pretraining datasets raises concerns about data quality and bias, necessitating further research into more efficient training strategies and robustness enhancements.
Future Work
Future directions include developing lightweight variants for edge deployment, exploring continual learning to adapt to new tasks without retraining from scratch, and integrating reinforcement learning for dynamic task adaptation. Further, enhancing interpretability and robustness, especially in real-world noisy scenarios, will be critical. Expanding the dataset diversity and exploring unsupervised domain adaptation could further improve generalization, making the model more applicable to real-time, large-scale video understanding in industry.
AI Executive Summary
The rapid growth of video content demands models capable of understanding complex, dynamic scenes across diverse tasks. Traditional approaches often excel in specific domains but lack generalization, limiting their practical utility. InternVideo addresses this challenge by unifying generative and discriminative self-supervised learning paradigms into a single, scalable framework. It leverages VideoMAE for masked spatiotemporal reconstruction and CLIP-based contrastive learning for semantic alignment, integrated through a learnable cross-modal attention module. This design enables the model to learn rich, multi-task representations from massive, multi-domain video datasets.
In extensive experiments, InternVideo achieved remarkable results, notably 91.1% top-1 accuracy on Kinetics-400, outperforming previous state-of-the-art methods by a significant margin. It demonstrated broad applicability across 39 datasets, covering tasks such as action recognition, detection, video-language retrieval, and question answering. The model’s architecture supports efficient large-scale pretraining, with modular components that facilitate scaling and adaptation. Its ability to dynamically fuse features from different training signals marks a key innovation, providing a new pathway for multi-task, multi-modal video AI.
This work has profound implications for both academia and industry. It paves the way for versatile, general-purpose video understanding systems capable of tackling real-world challenges. The approach’s scalability and transferability suggest promising future directions, including real-time applications, content creation, and intelligent surveillance. Despite its impressive performance, challenges remain in model size, computational cost, and robustness, which future research must address. Overall, InternVideo sets a new benchmark, inspiring ongoing innovation in the quest for truly intelligent video AI.
Deep Analysis
Background
视频理解作为计算机视觉的重要分支，经历了从传统特征工程到深度学习的快速演变。早期模型如C3D、I3D主要依赖空间和短时序信息，近年来，Transformer架构如VideoTransformer和VideoMAE引领了时空特征学习的革新。预训练模型如CLIP和VideoMAE极大提升了特征表达能力，但多任务、多模态的通用模型仍未成熟。现有方法多偏重于单一任务，缺乏跨任务的迁移能力，难以满足实际复杂场景的需求。随着视频内容的多样化和应用场景的扩大，迫切需要一种能在多个任务间迁移、融合不同模态信息的统一模型。
Core Problem
当前视频理解模型多为任务特定或单模态，难以实现多任务、多模态的泛化。高维视频数据带来计算挑战，模型在时空特征捕获与语义理解之间难以兼顾。如何设计一个既能高效预训练，又能跨任务迁移的通用模型，是行业和学术界面临的核心难题。此外，模型在复杂场景和细粒度动作识别中的鲁棒性不足，限制了其实际应用潜力。
Innovation
本研究提出融合VideoMAE的生成式掩码重建与CLIP的判别式对比学习的统一框架，创新性引入可学习的跨模态交互（CMA）模块，实现多模态特征的动态融合。架构采用ViT和UniformerV2，支持大规模预训练，显著提升时空建模能力。结合多任务训练策略，模型在动作识别、检测、视频问答等多个任务中实现SOTA，突破了单一任务导向的限制。该方法为多模态、多任务视频理解提供了新思路。
Methodology
- �� 利用VideoMAE进行视频掩码重建，输入高比例掩码视频片段，训练模型学习丰富的时空特征。• 采用CLIP基础的多模态对比学习，通过视频和文本的对齐，增强语义理解能力。• 构建两个不同结构的Transformer（ViT和UniformerV2）分别进行生成和判别训练。• 引入可学习的跨模态交互（CMA）模块，通过交叉注意力实现特征的动态融合。• 训练过程中，模型先独立优化两个分支，再通过CMA进行融合，最后在多任务上微调。
Experiments
采用39个公开视频数据集，涵盖动作识别、检测、视频问答等任务。模型在Kinetics-400达91.1%准确率，超越所有SOTA方法。通过消融实验验证掩码重建和对比学习的互补性，模型在多任务上表现优异。不同规模模型（如ViT-H、Large）在多个数据集上均获得显著提升。训练细节包括使用12M视频数据，采用AdamW优化，预训练时间超过两周，硬件配置为128个NVIDIA A100 GPU。
Results
在Kinetics-400上，InternVideo达91.1%的Top-1准确率，优于之前的MaskFeat、CoCa等方法。在Something-Something V2达到77.2%，在多任务场景中表现出色。模型在动作检测、视频问答、视频检索等任务中均实现了SOTA，验证了其广泛适应能力。消融分析显示，融合生成与判别特征比单一训练方案效果更佳，跨模态交互提升了多任务性能。
Applications
模型可广泛应用于视频内容检索、智能监控、自动视频标注、内容生成等场景。只需预训练模型和少量微调，即可实现多任务迁移，降低工业门槛。未来，结合边缘计算和硬件优化，模型有望在实时视频分析中发挥重要作用，推动智能视频内容产业的发展。
Limitations & Outlook
模型参数庞大，训练和部署成本高，限制在资源有限环境中的应用。极端复杂或长时序视频中的细粒度动作识别仍有提升空间。数据依赖性强，预训练数据质量直接影响性能，未来需探索更高效的模型压缩与自适应机制。
Plain Language Accessible to non-experts
想象你在一家大型工厂，工厂里有很多不同的机器和工人，他们每天都在完成各种任务。有些机器专门用来组装，有些用来检测产品质量。为了让工厂更高效，管理者希望所有机器都能理解彼此的工作内容，甚至能帮忙完成一些复杂的任务。InternVideo就像是给工厂配备了一个超级智能的管理系统，它可以学习每台机器的工作方式，理解不同工序之间的关系，还能在不同任务间切换。它通过观察大量工厂的操作录像，学习如何识别动作、理解指令、找到问题。这样，无论是检测产品缺陷、追踪生产流程，还是回答工厂管理者的问题，它都能胜任。这个系统的特别之处在于，它既能自己学习（生成式），也能理解别的机器传来的信息（判别式），还会不断调整自己，变得更聪明。未来，这样的系统可以让工厂变得更智能、更自动化，生产效率大大提高。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的老师和学生。有时候，老师会用录像来教你如何做数学题，有时候你需要用视频来解释一个故事。InternVideo就像是一个超级聪明的学习伙伴，它可以看很多视频，学会识别动作、理解内容，还能用文字描述视频里的场景。它既像是在自己练习（自己看视频做题），也像是在和老师或同学交流（理解文字和视频的关系）。这个伙伴通过学习很多不同类型的视频，变得非常聪明，不仅能帮你找到喜欢的视频，还能回答你关于视频的问题。它的特别之处在于，它可以同时学习两种技能：一是自己看视频猜动作，二是理解视频和文字的关系。这样，无论你是在找动作、理解故事，还是问问题，它都能帮上忙。未来，这样的伙伴会让我们看视频、学习、交流变得更方便、更有趣，就像拥有一个会看会说的超级助手一样！
Glossary
VideoMAE (Video Masked Autoencoder)
一种通过掩码部分视频帧，训练模型重建完整视频的自监督方法，提升时空特征学习能力。
用于视频掩码重建任务中。
Contrastive Learning (对比学习)
通过拉近相关样本的特征距离，推远无关样本，实现多模态语义对齐的自监督技术。
用于视频-文本对齐任务。
Cross-Model Attention (跨模态注意力)
在不同模型之间引入的注意力机制，用于动态融合多模态特征。
实现生成与判别特征的交互融合。
Vision Transformer (ViT)
基于Transformer架构的图像/视频特征提取模型，善于捕获长距离依赖。
作为基础编码器使用。
UniformerV2
结合局部与全局空间时间建模的Transformer变体，提升视频理解能力。
用于多模态对比学习中的视频编码。
Open Questions Unanswered questions from this research
- 1 如何进一步减少模型参数以适应边缘设备的部署需求，仍是未来的研究方向。
- 2 在极端复杂或长时序视频中的细粒度动作识别能力仍有待提升，特别是在实际应用中。
- 3 多模态融合的鲁棒性和解释性不足，未来需增强模型的透明度和可解释性。
Applications
Immediate Applications
视频内容检索
利用InternVideo实现多模态视频搜索，支持语音、文字、图像等多模态输入，提升检索效率和准确性。
智能监控
在监控场景中，快速识别异常行为或特定动作，增强安全性，减少人工监控成本。
Long-term Vision
智能内容生成
结合生成模型，实现自动视频剪辑、内容创作，推动娱乐和广告行业的变革。
Abstract
The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .