Masked Feature Prediction for Self-Supervised Visual Pre-Training
MaskFeat introduces feature prediction via masked regions, achieving 86.7% top-1 accuracy on Kinetics-400 without supervision, outperforming previous methods.
Key Findings
Methodology
This paper proposes Masked Feature Prediction (MaskFeat), a self-supervised pretraining framework where parts of the input video are randomly masked, and the model learns to regress the features of these masked regions. Using a Transformer backbone, the approach involves masking space-time cubes, then predicting target features such as Histogram of Oriented Gradients (HOG), deep network activations, or other descriptors. The training is unsupervised, relying solely on the features extracted from the original unmasked data as supervision. The study compares five feature types, finding HOG offers the best trade-off between performance and efficiency. The method is scalable, enabling large-scale training on unlabeled videos, and transfers effectively to image classification tasks, achieving competitive results on ImageNet.
Key Results
- Pretraining with MaskFeat on Kinetics-400 yields 86.7% top-1 accuracy with the MViT-L model, surpassing previous unsupervised methods by 5.2%. On Kinetics-600 and Kinetics-700, the models achieve 88.3% and 80.4%, respectively, demonstrating robust scalability. The approach also achieves 39.8 mAP on AVA action detection and 75.0% accuracy on SSv2. When extended to image classification, MaskFeat attains 84.0% top-1 accuracy on ImageNet with ViT-B, showing strong cross-modal transfer. These results highlight the method’s effectiveness in learning rich visual representations directly from unlabeled data.
- The study reveals that using HOG as a regression target significantly improves performance, owing to its ability to capture local shape and edge information while being invariant to illumination and contrast changes. The authors demonstrate that predicting continuous features, rather than discretized tokens or pixel values, strikes an optimal balance between accuracy and computational cost. Extensive ablation studies confirm that HOG-based MaskFeat outperforms alternatives such as dVAE codes, deep features from supervised models, and pseudo-labels, establishing it as the default choice for scalable pretraining.
- Experimental results across multiple datasets and model sizes confirm that MaskFeat pretraining leads to substantial performance gains without external supervision or large labeled datasets. The models trained with MaskFeat generalize well to downstream tasks, including action recognition, detection, and image classification, validating the approach’s broad applicability. The findings suggest that feature prediction is a promising direction for self-supervised learning in dense visual domains, especially when combined with scalable Transformer architectures.
- Overall, the results demonstrate that MaskFeat provides a simple yet powerful framework for self-supervised visual pretraining, opening new avenues for research and practical deployment in video understanding and beyond.
Significance
This work marks a significant advancement in self-supervised learning for dense visual data, particularly videos. By directly regressing rich local features like HOG, the method circumvents the need for external vocabularies or complex discretization, simplifying the pretraining pipeline. The ability to learn from unlabeled videos at scale addresses a critical bottleneck in the field—reducing reliance on costly annotations. The high performance on standard benchmarks underscores the potential for deploying such models in real-world applications like surveillance, autonomous driving, and content analysis, where labeled data is scarce or expensive. Moreover, the cross-modal transferability to image classification broadens the impact, suggesting a unified framework for visual representation learning across modalities.
Technical Contribution
The core technical innovation lies in formulating a feature regression task that directly predicts hand-crafted or learned features of masked regions, instead of relying on classification or contrastive objectives. The approach leverages a Transformer architecture with a simple masking strategy, inspired by BERT, but adapted for dense visual signals. It introduces the use of HOG as a target feature, which is computationally inexpensive and effective at capturing local shape information. The framework supports multiple feature types, including deep activations and pseudo-labels, but demonstrates that simple, classical features like HOG outperform more complex alternatives. The method’s scalability and simplicity make it a practical alternative to existing self-supervised methods, especially in large-scale video settings.
Novelty
This study is the first to systematically evaluate the effectiveness of predicting local handcrafted features, such as HOG, as a self-supervised pretraining objective for dense visual data. Unlike prior methods that depend on external tokenizers or discretized representations, MaskFeat employs continuous feature regression, simplifying the pipeline. The work also extends the concept from image to video domains seamlessly, demonstrating that local feature prediction can effectively capture complex spatiotemporal structures. Its novelty lies in combining classical computer vision descriptors with modern Transformer-based pretraining, leading to state-of-the-art results without external supervision or large labeled datasets.
Limitations
- While HOG provides a good balance of efficiency and performance, it may not fully capture complex semantic information, limiting the model's ability to understand high-level concepts. In scenes with extreme lighting or occlusion, the robustness of HOG-based prediction might degrade.
- The pretraining process remains computationally intensive, especially for large-scale video datasets, requiring significant hardware resources. The approach also relies on the quality of the extracted features, which may vary across different feature types.
- The current framework primarily focuses on dense local features; integrating higher-level semantic features or multi-modal signals could further improve performance but adds complexity. Additionally, the method’s effectiveness on very small datasets or domain-specific tasks needs further validation.
Future Work
Future research could explore incorporating multi-modal features, such as audio or textual cues, to enrich the learned representations. Developing adaptive feature extraction strategies that combine classical descriptors with learned features may enhance robustness. Reducing computational costs through model compression or more efficient masking strategies is another promising direction. Extending the framework to handle more complex tasks like video captioning or multi-object tracking, and investigating its applicability in low-data regimes, are also valuable avenues for future exploration.
AI Executive Summary
The rapid growth of video data has created both opportunities and challenges for artificial intelligence. While deep learning models, especially Transformer-based architectures, have achieved remarkable success in understanding visual content, their reliance on large labeled datasets remains a significant bottleneck. Annotating videos is labor-intensive and costly, limiting the scalability of supervised approaches. To address this, self-supervised learning has emerged as a promising paradigm, enabling models to learn from unlabeled data by defining proxy tasks that encourage meaningful feature extraction.
In this context, the paper introduces Masked Feature Prediction (MaskFeat), a novel self-supervised pretraining framework tailored for dense visual signals like videos. Inspired by masked language modeling in NLP, MaskFeat involves randomly masking out space-time regions within videos and training the model to predict the features of these regions. Unlike prior methods that rely on discretized tokens or external vocabularies, MaskFeat directly regresses continuous features, simplifying the pipeline and reducing dependency on external models.
The authors explore a broad spectrum of feature types, including raw pixel colors, hand-crafted descriptors like HOG, deep features from CNNs and ViTs, and pseudo-labels generated by external classifiers. Their extensive experiments reveal that HOG, a classical computer vision feature, strikes the best balance between efficiency and performance. By precomputing HOG features for entire videos and training the model to regress these features in masked regions, MaskFeat effectively captures local shape and edge information, which is crucial for understanding complex spatiotemporal patterns.
Empirical results demonstrate that models pretrained with MaskFeat achieve state-of-the-art performance on multiple benchmarks. On Kinetics-400, the pre-trained MViT-L model attains 86.7% top-1 accuracy, surpassing previous unsupervised methods by over 5%. Similar gains are observed on larger datasets like Kinetics-600 and Kinetics-700. The approach also generalizes to downstream tasks such as action detection (AVA) and human-object interaction classification (SSv2), with significant improvements. Moreover, extending MaskFeat to static images by treating a single frame as a video yields competitive ImageNet classification results, confirming its versatility.
This work offers a scalable, efficient, and effective framework for self-supervised visual pretraining. By leveraging simple local features, it reduces complexity and computational overhead, making it suitable for large-scale applications. Its success in both video and image domains paves the way for broader adoption of feature prediction strategies, reducing reliance on labeled data and opening new avenues for research in dense visual understanding. Overall, MaskFeat represents a significant step forward in unsupervised learning, with promising implications for future multi-modal and multi-task AI systems.
Deep Dive
Abstract
We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer-based models. Without using extra model weights or supervision, MaskFeat pre-trained on unlabeled videos achieves unprecedented results of 86.7% with MViT-L on Kinetics-400, 88.3% on Kinetics-600, 80.4% on Kinetics-700, 39.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageNet.
References (20)
The Kinetics Human Action Video Dataset
W. Kay, João Carreira, K. Simonyan et al.
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
M. Ryoo, A. Piergiovanni, Anurag Arnab et al.
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, R. Socher et al.
Histograms of oriented gradients for human detection
Navneet Dalal, B. Triggs
Object recognition from local scale-invariant features
D. Lowe
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee et al.
Swin Transformer V2: Scaling Up Capacity and Resolution
Ze Liu, Han Hu, Yutong Lin et al.
BEiT: BERT Pre-Training of Image Transformers
Hangbo Bao, Li Dong, Furu Wei
A Short Note about Kinetics-600
João Carreira, Eric Noland, Andras Banki-Horvath et al.
Emerging Properties in Self-Supervised Vision Transformers
Mathilde Caron, Hugo Touvron, Ishan Misra et al.
Multiscale Vision Transformers
Haoqi Fan, Bo Xiong, K. Mangalam et al.
An Empirical Study of Training Self-Supervised Vision Transformers
Xinlei Chen, Saining Xie, Kaiming He
SlowFast Networks for Video Recognition
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik et al.
Zero-Shot Text-to-Image Generation
A. Ramesh, Mikhail Pavlov, Gabriel Goh et al.
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Chen Sun, Abhinav Shrivastava, Saurabh Singh et al.
A Short Note on the Kinetics-700 Human Action Dataset
João Carreira, Eric Noland, Chloe Hillier et al.
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Priya Goyal, Piotr Dollár, Ross B. Girshick et al.
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection
Yanghao Li, Chaoxia Wu, Haoqi Fan et al.
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
João Carreira, Andrew Zisserman
Self-Attention with Relative Position Representations
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
Cited By (20)
BIMM: Brain-Inspired Masked Modeling for Video Representation Learning
Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining
LMVQ: Label-Free Metric-Learning for General AI-Generated Video Quality Assessment
Separators in Enhancing Autoregressive Pretraining for Vision Mamba
From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding
Learning Context-Adaptive Motion Priors for Masked Motion Diffusion Models with Efficient Kinematic Attention Aggregation
Global Cross-Modal Geo-Localization: A Million-Scale Dataset and a Physical Consistency Learning Framework
Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
A Two-stage Transformer Framework for Temporal Localization of Distracted Driver Behaviors
The Impact of Cervical Cytology Category Imbalance on Self-Supervised Representation Learning
Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
TrackMAE: Video Representation Learning via Track Mask and Predict
MEDiC: Multi-objective Exploration of Distillation from CLIP
AST-Adapter: Parameter-Efficient Video-to-Video Transfer Learning With Adaptive Spatiotemporal Information Bias
V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos
Co-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images
PolarMAE: Efficient Fetal Ultrasound Pre-training via Semantic Screening and Polar-Guided Masking
Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
A comprehensive survey of pretrained models for autonomous driving: Methods and applications
VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals