Back to the Features: DINO as a Foundation for Video World Models
Proposes DINO-world, a latent space video predictor trained on 60M videos, outperforming SOTA in prediction and physics understanding.
Key Findings
Methodology
This work introduces a video world model built upon a frozen DINOv2 image encoder, which maps video frames into a semantic feature space. A transformer-based predictor, trained with a smooth L1 loss on a large-scale dataset (~60M videos), predicts future features conditioned on past features and timestamps. The model handles variable frame rates and resolutions, enabling dense forecasting, physics understanding, and planning. Fine-tuning with observation-action trajectories allows action-conditioned control. The architecture emphasizes efficiency, with less than 1 billion parameters, surpassing pixel-based models like COSMOS in accuracy and resource use.
Key Results
- On dense segmentation forecasting (VSPW), the model achieves a 6.3% higher mIoU at 0.5s prediction horizon compared to the second-best. It also outperforms V-JEPA and COSMOS in depth and segmentation metrics. In physics benchmarks (IntPhys, GRASP), it shows comparable surprise scores to larger models, indicating solid physical reasoning. Fine-tuning on action trajectories improves planning in RL environments, demonstrating versatility.
- The model's performance scales with data and model size; larger transformers and more diverse datasets yield better results. Ablation studies reveal that pretraining on 60M videos significantly boosts generalization. The approach excels in multi-task settings, including dense prediction, physics, and control, validating the efficiency of latent feature prediction over pixel-level generation.
- Comparison with baselines shows that the proposed method achieves superior accuracy with much lower computational cost. It surpasses pixel-based models like COSMOS in scene dynamics modeling and generalizes well across datasets, confirming the effectiveness of leveraging semantic features for future prediction.
Significance
This research advances video understanding by leveraging large-scale self-supervised image models to build efficient, generalist world models. It addresses core limitations of pixel-based models—computational cost, limited generalization—by operating in a semantic feature space. The ability to predict future scene features accurately supports applications in autonomous driving, robotics, and virtual environments, enabling better planning, control, and physics reasoning. The framework's scalability and transferability mark a significant step toward unsupervised, large-scale video intelligence, bridging the gap between perception and action in complex, real-world scenarios.
Technical Contribution
The paper introduces a transformer-based auto-regressive predictor operating in the semantic feature space of a frozen DINOv2 encoder. It employs cross-attention mechanisms for flexible, multi-scale future prediction, supporting variable frame rates and resolutions. The training leverages a large, uncurated video dataset, reducing model size (<1B parameters) and computational costs compared to pixel-based counterparts like COSMOS. The addition of observation-action fine-tuning enables action-conditioned planning, making the model adaptable for control tasks. This architecture offers a new paradigm for scalable, multi-task video modeling, combining efficiency with high accuracy.
Novelty
This is the first large-scale, self-supervised video prediction model operating solely in a pre-trained semantic feature space, avoiding pixel-level complexity. Unlike prior pixel-based generative models, it leverages the semantic richness of DINOv2 features, enabling efficient, accurate future scene prediction. Its transformer architecture supports variable temporal horizons and multi-task learning, bridging the gap between scene understanding and control, a novel approach in the domain.
Limitations
- Dependence on the quality of pre-trained features; in scenarios with domain shift or low semantic richness, performance may degrade. Despite efficiency gains, training still requires substantial compute resources, especially for very large models. The model's physics understanding, while strong, remains limited in highly complex or chaotic environments, necessitating further integration of explicit physical reasoning modules.
Future Work
Future directions include integrating multi-modal inputs such as language and audio, enhancing physical reasoning with explicit models, and improving transfer to real-world robotic systems. Exploring continual learning to adapt to new environments without retraining from scratch, and extending the framework to multi-agent scenarios, are also promising avenues. Further research aims to reduce computational costs and increase robustness across diverse, real-world conditions.
AI Executive Summary
The explosion of video data has created a pressing need for models that can efficiently understand and predict dynamic scenes. Traditional pixel-level generative models, while capable of producing high-fidelity videos, are computationally intensive and often domain-limited. Addressing this challenge, the present work introduces DINO-world, a novel approach that operates in the semantic feature space derived from a pre-trained DINOv2 image encoder. This paradigm shift enables scalable, resource-efficient future prediction across diverse scenes.
At the core, DINO-world employs a transformer-based predictor trained on approximately 60 million uncurated videos, capturing the temporal dynamics of complex environments. Unlike pixel-based models, it predicts future scene features, which can be used for dense forecasting, physics reasoning, and planning. The model handles variable frame rates and resolutions, making it adaptable to real-world scenarios. Experimental results demonstrate that DINO-world surpasses existing models like V-JEPA and COSMOS in accuracy, with a 6.3% improvement in segmentation mIoU at 0.5 seconds, and maintains strong physics understanding.
A key innovation lies in leveraging large-scale self-supervised image models, reducing computational costs and enhancing generalization. Fine-tuning with observation-action trajectories further enables control applications, validated through reinforcement learning benchmarks. This approach bridges perception and action, offering a unified framework for scene understanding and planning.
The broader impact of this work is significant: it paves the way for autonomous agents that can learn from vast unlabeled video data, perform reliable scene prediction, and plan actions efficiently. While promising, challenges remain in modeling highly complex physical interactions and reducing training costs further. Future research will focus on multi-modal integration, explicit physics modeling, and real-world deployment, aiming to realize truly autonomous, adaptable intelligent systems.
Deep Dive
Glossary
DINOv2 (DINO version 2)
一种自监督学习的视觉变换器模型，提取丰富的图像语义特征，用于下游任务。
作为视频帧的编码器，将每帧映射到潜在特征空间。
Transformer (变换器)
一种基于注意力机制的深度学习架构，擅长处理序列数据，用于预测未来场景特征。
构建未来预测器的核心架构。
Auto-regressive Model (自回归模型)
一种逐步预测未来状态的模型，利用已知信息递推生成序列。
本文中的未来帧特征预测机制。
Smooth L1 Loss (平滑L1损失)
一种结合了L1和L2优点的损失函数，用于训练预测模型，减少异常值影响。
优化未来特征预测的目标函数。
Uncurated Video Dataset (未标注视频集)
未经过人工标注、随机采集的大规模视频数据，用于模型预训练。
训练模型的主要数据来源。
Open Questions Unanswered questions from this research
- 1 尽管模型在大规模数据上表现优异，但在极端物理场景或细粒度预测方面仍有限。如何结合显式物理模型以增强推理能力，仍是未来研究重点。
- 2 模型在多模态融合方面尚未充分探索，结合声音、文本等信息，可能进一步提升场景理解和预测能力。
- 3 实现模型在实际机器人和自主系统中的无缝迁移，仍面临域适应和样本效率的挑战。
Applications
Immediate Applications
自动驾驶辅助
利用模型预测未来道路场景变化，增强车辆的路径规划和避障能力，提升行驶安全性。
机器人导航与控制
在机器人自主导航中，预测环境动态，支持实时路径规划和动作决策，减少对传感器依赖。
Long-term Vision
自主智能体
实现具有自主学习和规划能力的智能系统，能在未知环境中自主适应和行动，推动智能机器人和虚拟助手的发展。
Abstract
We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2. By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes, from driving and indoor scenes to simulated environments. We show that DINO-world outperforms previous models on a variety of video prediction benchmarks, e.g. segmentation and depth forecasting, and demonstrates strong understanding of intuitive physics. Furthermore, we show that it is possible to fine-tune the predictor on observation-action trajectories. The resulting action-conditioned world model can be used for planning by simulating candidate trajectories in latent space.