A Survey: Spatiotemporal Consistency in Video Generation
This survey frames video generation as sequential sampling from a high-dimensional spatiotemporal distribution and compares VAE, AR, diffusion, and flow models.
Key Findings
Methodology
The paper represents a video as V=(x₁,…,x_T), where x_t may be a frame, local patch, or latent feature block, and treats generation as conditional sequential sampling under text, image, or other control C. Spatial consistency covers identity, layout, lighting, color, texture, and semantics; temporal consistency covers smooth motion, state transitions, flicker suppression, and action semantics. The survey organizes methods across models, representations, frameworks, post-processing, training, and evaluation.
Key Results
- The comparison table characterizes autoregressive and diffusion models as generally stronger for spatiotemporal and long-term consistency. AR models benefit from causal conditioning, while diffusion models obtain strong practical quality through iterative denoising and global refinement. VAEs are mainly useful for compression and reconstruction; flow models offer smooth local trajectories but weaker long-range dependency modeling.
- 3D-VAE, OD-VAE, 3D Causal VAE, VideoMAE, MotionAura, CogVideoX, and HunyuanVideo improve latent representation through spatiotemporal compression, causal encoding, masked reconstruction, or dynamic modeling. The supplied text contains no unified benchmark scores, so specific percentage gains cannot be responsibly inferred.
- Diffusion uses the noise-prediction objective L=E||ε−εθ(x,t)||², while flow matching uses L=E||vθ(x,t)−(x₁−x₀)||². Pyramid-Flow uses feature pyramids for stable multiscale flows; FlashVideo uses an approximately linear sampling trajectory to combine efficiency, detail, and consistency.
Significance
This survey unifies failures such as flicker, identity drift, background switching, and object teleportation as compatibility and transition problems in a spatiotemporal distribution. It connects architecture, latent representation, sampling, training, and evaluation, helping researchers select methods by failure mode rather than by single-frame sharpness. For industry, the framework explains why systems such as Sora, Veo, and Runway Gen must jointly optimize visual quality and cross-frame stability. It also shows why image metrics alone are inadequate for video generation.
Technical Contribution
The main contribution is a consistency-oriented taxonomy rather than a new generator. The paper separates spatial consistency from temporal consistency and formalizes the latter through p(x_t|x_<t,C). It compares four generation routes—VAE, AR, diffusion, and flow—and organizes representation strategies including compression, long-sequence encoding, discretization, feature decoupling, and diffusion-timestep features. It also covers interpolation, super-resolution, stabilization, deblurring, transfer learning, progressive learning, joint image-video learning, reward feedback, distillation, and training-free correction.
Novelty
Unlike surveys organized primarily by model family or application, this work uses sequential sampling from a high-dimensional spatiotemporal distribution as its central lens. Its novelty lies in integrating models, representations, generation frameworks, post-processing, training, benchmarks, and metrics around consistency mechanisms. It does not introduce a new loss, dataset, or experimentally validated state-of-the-art generator; its contribution is conceptual coverage and structured synthesis.
Limitations
- As a survey rather than a new method paper, it does not provide a controlled experimental table with common datasets, identical compute, or numerical attribution of improvements to identity, motion, and flicker mechanisms.
- Consistency is task-dependent and partly subjective. Existing metrics often isolate semantics, visual quality, or motion, leaving long-term dynamics, physical plausibility, and condition adherence without one agreed evaluation protocol.
Future Work
Future work should build standardized benchmarks spanning identity, layout, lighting, motion, physical plausibility, and dynamic semantics. Promising systems may combine AR’s causal memory, diffusion’s global refinement, and flow models’ efficient trajectories. Important engineering goals include longer videos, lower memory, real-time inference, interactive control, training-free repair, and evaluation calibrated to human preferences and physical constraints.
AI Executive Summary
Video generation has moved beyond producing attractive individual frames, yet a convincing sequence must preserve the same subject, setting, lighting, and action over time. Identity swaps, background changes, color flicker, and object teleportation expose the weakness of image-centric evaluation. This survey recasts the task as sequential sampling from a high-dimensional spatiotemporal distribution, giving a common language for these failures.
The authors compare four model families. Autoregressive models generate each unit from previous units through p(x_i|x_<i;θ), providing causal temporal structure but slow inference. Diffusion models learn to reverse a noise process using L=E||ε−εθ(x,t)||² and offer strong practical quality through iterative global refinement. Flow models learn smooth, reversible trajectories with flow matching, while VAEs primarily provide compact latent representations. 3D Causal VAE, VideoMAE, Pyramid-Flow, and FlashVideo illustrate complementary solutions at the representation and sampling levels.
The survey also maps conditional, multistage, and interactive frameworks; interpolation, super-resolution, stabilization, and deblurring; and transfer, progressive, joint image-video, reward-feedback, distillation, and training-free strategies. Its value is a research map rather than a new numerical record. The supplied text reports no unified datasets or benchmark scores, so conclusions should be read mechanistically: diffusion and AR are broadly strong, but long-horizon dynamics, standardized evaluation, efficiency, controllability, and physical realism remain unresolved.
Deep Analysis
Background
AIGC has accelerated products such as Runway’s Gen series, Google DeepMind’s Veo, and OpenAI’s Sora. Unlike images, videos combine spatial structure with temporal evolution. Diffusion, autoregressive, flow, and 3D latent models have advanced quality, but relevant work is fragmented across generation, compression, training, and post-processing. The survey addresses the missing consistency-centered synthesis.
Core Problem
The target is spatial compatibility of identity, layout, style, lighting, color, texture, and semantics, together with temporally smooth motion, valid state transitions, low flicker, and faithful action semantics. Formally, generation must model p(x_t|x_<t,C). This is difficult because local errors accumulate, long-range dependencies are expensive, and independent frame quality does not guarantee coherent dynamics.
Innovation
First, consistency is reformulated as sequential sampling from a high-dimensional spatiotemporal distribution. Second, the paper proposes a six-part map covering models, representations, frameworks, post-processing, training, and evaluation. Third, it links failures to mechanisms: causal representations constrain temporal information flow, compression reduces redundancy, iterative denoising enables global correction, and post-processing repairs local artifacts.
Methodology
- �� Models: VAE uses p(x)=∫p(z)p(x|z)dz for probabilistic compression; AR applies maximum likelihood to sequential units; diffusion predicts noise with L=E||ε−εθ(x,t)||²; flow models learn continuous paths with flow matching.
- �� Representations: 3D-VAE and OD-VAE jointly encode space and time; 3D Causal VAE restricts encoding to current and previous information; VideoMAE learns contextual structure by masked reconstruction; LARP uses learnable queries for holistic video tokenization.
- �� Frameworks and repair: conditional, multistage, and interactive generation are combined with interpolation, video super-resolution, stabilization, and deblurring. Training strategies include transfer, progressive, joint image-video, reward-feedback, distillation, and training-free methods.
Experiments
The supplied sections describe a literature survey and qualitative comparison. Table 2 compares VAE, AR, diffusion, and flow by consistency, long-term consistency, quality, controllability, training stability, data requirements, speed, memory, and disadvantages. The provided text does not include a unified dataset list, baseline scores, hyperparameters, ablations, FVD, CLIP, or user-study numbers; these cannot be fabricated.
Results
The mechanistic conclusion is that AR’s causal structure supports ordered and long-term consistency, while diffusion achieves strong practical quality through iterative global optimization. VAE is valuable for stable compression and reconstruction; flow models offer local smoothness and few-step sampling but struggle with complex long-term dependencies. Pyramid-Flow and FlashVideo are cited as representative advances, without numerical gains in the supplied text.
Applications
The taxonomy supports text-to-video, image-to-video, video extension, controllable editing, advertising, film previsualization, and game content. Deployment requires an appropriate 3D latent representation, long-sequence mechanism, and conditioning framework, followed by artifact repair when needed. Identity, motion, and style continuity are especially important in entertainment and commercial media.
Limitations & Outlook
The survey lacks controlled quantitative comparisons under shared datasets and compute. Metrics often separate semantics, motion, and visual quality. Long videos amplify error accumulation, memory use, and sampling cost; AR is slow, diffusion is expensive, and flow models face long-range complexity. Future systems need multidimensional benchmarks, physical and causal evaluation, long-context memory, efficient sampling, and interactive correction.
Plain Language Accessible to non-experts
Imagine a factory making an animated film. Each frame is one product on a conveyor belt. A single product can look beautiful, but the whole batch fails if the character changes clothes without explanation, the room suddenly becomes a forest, or the lights flicker. The factory therefore needs memory: workers must know what was made before and what should happen next.
In the paper’s picture, a VAE is the warehouse that compresses huge video material into manageable packages. An autoregressive model is the assembly line that checks earlier products before making the next one. A diffusion model is a quality-control team that repeatedly cleans and corrects a rough draft. A flow model is a smooth rail carrying a rough version toward a detailed version. A causal 3D representation prevents workers from using information from the future.
After production, interpolation fills missing moments, super-resolution sharpens images, stabilization reduces shaking, and deblurring repairs motion damage. The survey’s message is that no single department solves consistency. Reliable video requires compact storage, temporal memory, controlled production, global inspection, local repair, and meaningful testing together.
ELI14 Explained like you're 14
Suppose you ask a computer to make a game clip of a hero running. In frame one the hero wears red, in frame two the outfit turns blue, and in frame three the street becomes a beach. Each picture might look fine alone, but the clip feels broken. The hard part of video generation is making every picture a believable continuation of the previous one.
The paper compares several kinds of digital directors. An autoregressive model writes the next scene after reading earlier scenes. A diffusion model starts with visual “static” and cleans it step by step. A flow model moves from a rough clip to a polished clip along a smooth path. A VAE is like shrinking a giant game map into a smaller file so the computer can handle it.
Methods such as 3D Causal VAE help the model remember the past without cheating by seeing the future. VideoMAE hides pieces and makes the model guess them, teaching it to use context. Pyramid-Flow and FlashVideo try to make generation smoother or faster. Interpolation and super-resolution are like adding missing snapshots and improving blurry ones.
The survey does not give one shared scoreboard with exact percentage wins in the supplied text, so we should not invent a winner. The remaining challenge is long video: characters, objects, actions, physics, and lighting must stay believable. The dream is a director that remembers, plans, checks, and fixes its own scenes!
Glossary
Spatiotemporal Consistency
The preservation of compatible appearance and meaning across space and time. It includes identity, layout, lighting, texture, motion, and dynamic semantics.
This is the survey’s organizing concept for reviewing models, representations, training, and evaluation.
Autoregressive Model
A model that predicts the next unit conditioned on previously generated units, commonly p(x_i|x_<i;θ). Its causal structure naturally represents ordered sequences.
The survey presents AR as strong for temporal and long-term consistency but slow at inference.
Diffusion Model
A model that learns to reverse gradual corruption by noise. A common objective is E||ε−εθ(x,t)||².
It is discussed as a high-quality route whose iterative denoising performs global refinement.
3D Causal VAE
A variational encoder-decoder that compresses spatial and temporal information while restricting encoding to current and past frames. This reduces future-information leakage.
CogVideoX and HunyuanVideo are cited as systems using this representation.
Flow Matching
A method that learns a continuous velocity field between a simple prior and data distribution, often using E||vθ(x,t)−(x₁−x₀)||². It enables smooth and potentially few-step sampling.
It underlies the survey’s discussion of flow models, Pyramid-Flow, and FlashVideo.
VideoMAE
A self-supervised method that masks spatiotemporal video regions and reconstructs them from context. This encourages learning motion and structural relationships.
The survey classifies it as a masked spatiotemporal representation approach.
Open Questions Unanswered questions from this research
- 1 How can one metric jointly evaluate identity, physical motion, dynamic semantics, and long-horizon stability? Existing metrics cover only subsets, motivating multidimensional benchmarks calibrated to human judgment.
- 2 How can systems avoid error accumulation in long videos while retaining real-time speed and low memory? Causal memory, global planning, fast sampling, and verifiable physical constraints must be combined.
- 3 How can textual action intent become a measurable temporal trajectory? Progress likely requires action-level annotations, causal evaluation, and interactive correction.
Applications
Immediate Applications
Advertising and short-form video
Content teams can use diffusion or flow generation for drafts, then apply stabilization, interpolation, and super-resolution to reduce flicker and improve clarity. They need text or reference-image controls and should manually verify identity, brand colors, action continuity, and semantic compliance.
Film and game previsualization
Artists can use 3D Causal VAE-based representations and conditional frameworks to explore shots, characters, and environments quickly. The result is suitable for storyboarding and iteration, but complex contact, collision, hand motion, and exact camera continuity may still require animation or physics-based correction.
Long-term Vision
Interactive world simulators
Long-sequence representations, causal memory, and dynamic-semantic evaluation could support explorable game or virtual-reality worlds. Major barriers include long-term consistency, physical realism, low-latency interaction, data scale, and copyright-safe training material.
Abstract
Video generation aims to produce temporally coherent sequences of visual frames, representing a pivotal advancement in Artificial Intelligence Generated Content (AIGC). Compared to static image generation, video generation poses unique challenges: it demands not only high-quality individual frames but also strong temporal coherence to ensure consistency throughout the spatiotemporal sequence. Although research addressing spatiotemporal consistency in video generation has increased in recent years, systematic reviews focusing on this core issue remain relatively scarce. To fill this gap, this paper views the video generation task as a sequential sampling process from a high-dimensional spatiotemporal distribution, and further discusses spatiotemporal consistency. We provide a systematic review of the latest advancements in the field. The content spans multiple dimensions including generation models, feature representations, generation frameworks, post-processing techniques, training strategies, benchmarks and evaluation metrics, with a particular focus on the mechanisms and effectiveness of various methods in maintaining spatiotemporal consistency. Finally, this paper explores future research directions and potential challenges in this field, aiming to provide valuable insights for advancing video generation technology. The project link is https://github.com/Yin-Z-Y/A-Survey-Spatiotemporal-Consistency-in-Video-Generation.