Diffusion Models for Video Prediction and Infilling
RaMViD uses random-mask diffusion and reaches SOTA on BAIR/Kinetics-600.
Key Findings
Methodology
RaMViD (Random-Mask Video Diffusion) adapts image diffusion to videos by replacing 2D convolutions with 3D convolutions in a U-Net and by changing how conditioning is injected. During training, only a random subset of frames U is noised; the conditioning frames C stay clean and are provided in the same tensor, and the loss is computed only on U, as in Eq. (8). A Bernoulli variable with probability pU can drop all conditioning (C=∅), allowing mixed conditional/unconditional training in one architecture. The mask size k is sampled uniformly up to K, so the model learns arbitrary completion patterns.
Key Results
- On BAIR, the model achieves state-of-the-art video prediction: FVD 85.3±1.8 with pU=0.25, K=4, beating FitVid (93.6), MCVD (89.5), and NÜWA (86.9); with K=8 it improves further to 82.64. This is for predicting 15 future frames from one observed frame.
- On Kinetics-600, predicting 11 frames from 5 observed frames yields FVD 17.53±1.07 (pU=0.25), far better than TrIVD-GAN-FP (26±1), CCVS (55±1), DVD-GAN-FP (69±1), and Video Transformer (170±5). For infilling with C={0,5,10,15}, it reaches FVD 4.91 (pU=0.25).
- For unconditional generation on UCF-101, RaMViD obtains IS 21.71±0.21 (pU=0.75) and 20.84±0.08 (pU=0.5). It is competitive with early GAN/video baselines such as MoCoGAN (12.42), TGAN-F (13.62), and TGANv2 (26.60), though below high-resolution DVD-GAN (32.97). Ablations show that large pU can hurt complex-data prediction, especially on Kinetics-600.
Significance
The paper demonstrates that diffusion models are not limited to static image synthesis; they can also serve as strong generative engines for temporally coherent video completion. Scientifically, this matters because video prediction and infilling require both realism and cross-frame consistency, a setting where many autoregressive, VAE, and GAN approaches struggle with blur, drift, or harmonization failures. Practically, the method provides a single framework for future-frame prediction, middle-frame infilling, and temporal upsampling, which is attractive for robotics, autonomous driving, video editing, and simulation. The key advance is not merely better scores, but a much simpler conditioning recipe that preserves the standard diffusion backbone and reduces the need for expensive resampling.
Technical Contribution
The main technical contribution is a randomized masking training scheme that makes conditioning implicit rather than architectural. Instead of feeding condition frames through a separate encoder or concatenation branch, RaMViD passes the whole video through the network while corrupting only the unknown subset. This keeps the same U-Net used for unconditional diffusion, enabling joint training with pU controlling the unconditional branch. The model uses a discrete diffusion process with a linear noise schedule, 3D convolutional residual blocks, and self-attention at 16 and 8 resolution. Compared with prior conditional diffusion systems, this design avoids mismatched input formats, removes the need for iterative re-sampling to harmonize known and unknown regions, and naturally supports arbitrary conditioning subsets for infilling and upsampling.
Novelty
RaMViD is among the first diffusion-based methods to unify video prediction, infilling, and temporal upsampling through one random-mask conditioning mechanism. The novelty is not just “diffusion for video,” but a training strategy that lets the same network handle both conditional and unconditional generation without extra modules. Relative to methods that concatenate conditions or require task-specific architectures, the paper’s core idea is to encode the task in the mask pattern itself.
Limitations
- Performance degrades when the unconditional rate pU becomes too large on complex data. On Kinetics-600, pU=0.75 raises prediction FVD to 27.64, suggesting that unconditional and conditional objectives can conflict when the data distribution is highly diverse.
- Fast motion remains difficult: the qualitative results show deformation in rapidly moving objects, especially on Kinetics-600 and UCF-101. This indicates that 3D convolutional diffusion still struggles to model abrupt large-displacement dynamics perfectly.
- Autoregressive rollout deteriorates over time. The authors report that quality compounds downward and often becomes poor after about 30 frames, so long-horizon generation remains an open challenge.
Future Work
The authors explicitly point to more task-specific masking schedules as an important next step, since different pU/K choices behave differently across datasets. They also suggest attaching super-resolution modules to improve high-resolution outputs. Beyond that, the random-mask idea could be tested on other sequence domains such as pose, robotics, or medical time series, where missing-data completion and temporal coherence are equally important.
AI Executive Summary
Deep Dive
Abstract
Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain. We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces a new conditioning technique during training. By varying the mask we condition on, the model is able to perform video prediction, infilling, and upsampling. Due to our simple conditioning scheme, we can utilize the same architecture as used for unconditional training, which allows us to train the model in a conditional and unconditional fashion at the same time. We evaluate RaMViD on two benchmark datasets for video prediction, on which we achieve state-of-the-art results, and one for video generation. High-resolution videos are provided at https://sites.google.com/view/video-diffusion-prediction.