LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
LeVJEPA uses collapse-free invariance loss and SIGReg regularization, reducing pretraining compute by up to 20x while maintaining or surpassing state-of-the-art performance.
Key Findings
Methodology
LeVJEPA employs a single encoder trained with an invariance loss between global and local views of a video clip, regularized by SIGReg to prevent collapse. The architecture comprises only an encoder and a projector, eliminating the need for target encoders, predictors, or stop-gradient operations. During training, random token dropping reduces computational load and enhances robustness. The model uses block-causal attention, ensuring representations are causal and suitable for streaming or autoregressive tasks. The hyperparameter is fixed, simplifying the training process. This approach allows for efficient pretraining with fewer tokens observed, leading to significant computational savings while maintaining high downstream accuracy.
Key Results
- At matched epochs and data, LeVJEPA achieves comparable or superior accuracy to V-JEPA 2 across ViT-S/B/L models, with 5.6 to 20.8 times less total pretraining compute. On ImageNet-1K, it improves linear probing accuracy by 7.6 points. Random token dropping not only reduces cost but also improves downstream performance, demonstrating efficiency gains.
- Under equal FLOP budgets, LeVJEPA outperforms the strongest video baseline by 7.6 points on ImageNet-1K and achieves the highest accuracy on Kinetics-400. It also approaches image-pretrained encoders in appearance-centric tasks while nearly doubling motion-centric accuracy, showing versatility.
- Using block-causal attention, the encoder models temporal causality directly, enabling incremental frame extension without re-encoding. This aligns with autoregressive modeling principles, supporting streaming inference and real-time applications.
Significance
This work addresses the high computational expense of video pretraining by proposing a simple, collapse-free framework that achieves state-of-the-art results with significantly reduced compute. It demonstrates that effective video representations can be learned without architectural asymmetries or auxiliary modules, making large-scale pretraining more accessible. The causal design aligns with predictive coding theories and supports real-time, streaming video understanding, opening new avenues for applications in autonomous systems, video analysis, and robotics. The approach also simplifies the training pipeline, lowering barriers for broader adoption and further research.
Technical Contribution
LeVJEPA introduces a collapse-free objective based on invariance loss combined with SIGReg regularization, removing the need for target networks, predictors, or stop-gradient mechanisms common in prior methods. Its architecture is minimalistic, comprising only an encoder and a projector, with a fixed hyperparameter. Random token dropping acts as both an efficiency measure and a data augmentation, improving downstream accuracy. The use of block-causal attention ensures temporal causality, enabling incremental inference. Extensive experiments validate its efficiency and robustness across multiple scales and tasks, setting new standards for scalable video pretraining.
Novelty
This is the first approach to entirely eliminate architectural asymmetries and auxiliary modules in video self-supervised learning, relying solely on a collapse-free invariance loss and SIGReg regularization. Unlike prior methods such as V-JEPA and VideoMAE, which depend on target encoders or masked pixel reconstruction, LeVJEPA simplifies the pipeline drastically. Its causal attention design and token dropping strategies further distinguish it, enabling efficient, streaming-compatible video representation learning with theoretical guarantees against collapse.
Limitations
- While highly efficient, the method still requires large-scale unlabeled video datasets and significant computational resources for pretraining, albeit less than traditional approaches.
- Extreme token sparsity may impair dynamic information capture, especially in highly motion-dependent tasks, indicating a trade-off between efficiency and dynamic fidelity.
- The current design focuses on short-term causality; modeling long-term dependencies remains a challenge and warrants future integration with more sophisticated temporal modules.
Future Work
Future directions include developing adaptive token dropping strategies to balance efficiency and dynamic information retention, extending the causal modeling to longer sequences, and integrating multimodal data for richer representations. Additionally, applying LeVJEPA to real-time video streaming, robotics, and video generation tasks could further demonstrate its practical impact. Exploring hybrid architectures that combine causal and bidirectional attention for complex scenarios is also promising.
AI Executive Summary
Video is a rich source of information about the physical world, capturing motion, causality, and object permanence. However, learning effective representations from video has traditionally been computationally expensive, requiring complex architectures with target encoders, predictors, and masking schemes. These methods, while effective, pose significant barriers to scaling and practical deployment.
This paper introduces LeVJEPA, a novel framework that simplifies video pretraining by removing the need for architectural asymmetries and auxiliary modules. The core idea is to train a single encoder using a collapse-free invariance loss, regularized by SIGReg, which guarantees the avoidance of trivial solutions. During training, random token dropping reduces the computational load, while the use of block-causal attention ensures the model captures temporal causality directly, enabling incremental frame extension.
Experimental results demonstrate that LeVJEPA achieves comparable or better downstream performance than state-of-the-art methods like V-JEPA 2, at a fraction of the computational cost—up to 20 times less pretraining compute. Notably, it surpasses strong baselines on ImageNet-1K and Kinetics-400, and approaches image-pretrained encoders in appearance tasks while nearly doubling motion understanding accuracy. These findings suggest that video, once computationally feasible, is an ideal substrate for general-purpose visual pretraining.
The approach’s simplicity, efficiency, and theoretical guarantees mark a significant step forward in scalable video representation learning. It opens new avenues for real-time applications, streaming inference, and broader accessibility, paving the way for future innovations in autonomous systems, video analysis, and multimodal AI. Despite its strengths, challenges remain in modeling long-term dependencies and further reducing computational costs, which will guide ongoing research efforts.
Deep Dive
Abstract
Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.