UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation
UniTemp enables bidirectional video generation via joint distillation, addressing boundary flickering caused by causal 3D VAE, with strong performance on short and long videos.
Key Findings
Methodology
This paper introduces UniTemp, a unified framework combining blockwise anchor latents with bidirectional distillation. It trains a single autoregressive student model capable of generating videos conditioned on past, future, or both, by jointly optimizing forward and backward tasks. The model leverages shared parameters, full-sequence attention, and auxiliary anchor latents to stabilize reverse generation. The training process distills a pre-trained teacher diffusion model, enabling efficient inference. The core innovation lies in restoring missing context at block boundaries during backward generation, significantly reducing flickering artifacts. Extensive experiments on short, long, and inbetween tasks demonstrate that UniTemp maintains competitive quality while offering versatile applications like scene transitions, looping, and bidirectional extension.
Key Results
- On short video tasks, UniTemp achieves a VBench score of 83.5, comparable to forward-only models, with boundary flickering reduced from a FR of 1.42 to 1.08. In long video generation, FVD improves by 15%, and scene transition quality is enhanced. Ablation shows anchor latents reduce flickering artifacts significantly, and shared models lower training costs. The model supports inbetween generation with flexible attention horizons, enabling smooth content interpolation.
- In long videos (up to 100s), UniTemp with sink latents maintains content stability, with a 10-15% reduction in flickering ratios and improved scene coherence. It performs well across diverse scenarios, including scene transitions and looping, demonstrating robustness and adaptability. The model's inference speed is comparable to traditional unidirectional models, thanks to caching and shared parameters.
- Ablation studies confirm that anchor latents are critical for reverse boundary stability, and shared training reduces complexity. Results indicate the approach effectively balances quality, efficiency, and flexibility, outperforming prior methods that support only forward generation.
Significance
This work fundamentally advances video synthesis by enabling models to generate content in any temporal order, greatly enhancing editing flexibility. It addresses the longstanding challenge posed by causal encoders, allowing for seamless bidirectional editing, scene manipulation, and long-horizon content creation. The unified model simplifies deployment, reduces computational costs, and opens new avenues for creative workflows in film, gaming, and virtual reality industries. By bridging the gap between high-quality generation and flexible editing, UniTemp sets a new standard in generative video modeling.
Technical Contribution
The main technical innovation is the introduction of blockwise anchor latents, which restore missing past context during reverse generation. This mechanism, combined with shared autoregressive training and full-sequence attention, stabilizes bidirectional generation without retraining the VAE. The framework leverages distillation from a pre-trained diffusion teacher, enabling efficient multi-directional inference. The unified model design reduces complexity and supports diverse applications, including inbetween filling and scene transitions. The approach provides theoretical guarantees for boundary stability and practical improvements in visual quality.
Novelty
UniTemp is the first to support flexible, bidirectional autoregressive video generation within a single model, overcoming the causal dependency limitations of traditional VAEs. The use of blockwise anchor latents to stabilize reverse boundary transitions is a novel contribution, enabling high-quality inbetween and scene editing tasks. Unlike prior methods limited to unidirectional generation, this approach offers a unified, efficient solution for diverse content creation workflows, representing a significant leap forward in generative modeling.
Limitations
- Despite improvements, the model may still struggle with extremely high-motion scenes, where residual artifacts or blurring occur due to limited latent expressiveness.
- Dependence on pre-trained teacher models introduces potential biases and limits adaptation to new domains without retraining.
- Handling ultra-long videos (>100s) or highly complex scenes remains challenging, requiring further optimization of anchor latents and attention mechanisms.
Future Work
Future research will focus on enhancing latent expressiveness, integrating multi-modal inputs (text, audio), and developing adaptive anchor strategies for better boundary stabilization. Efforts will also aim to reduce computational costs further and extend the framework to real-time applications. Exploring unsupervised or weakly supervised training paradigms could improve domain adaptation, and incorporating user-guided editing tools will make the technology more accessible for creative industries.
AI Executive Summary
Recent advances in video generation have been driven by diffusion models and transformer architectures, achieving remarkable visual quality. However, these models are predominantly unidirectional, limiting their flexibility for content editing and scene manipulation. In practical workflows, creators often need to extend videos backward, fill in gaps, or generate intermediate frames, which existing models cannot efficiently support due to their causal structure. This bottleneck restricts creative freedom and complicates long-horizon content synthesis.
To address this, the authors propose UniTemp, a novel framework that unifies bidirectional video generation within a single autoregressive model. The key challenge lies in overcoming the boundary discontinuities caused by the causal 3D VAE, which encodes latents conditioned solely on past context. The solution involves introducing blockwise anchor latents—auxiliary variables that restore missing past information at block boundaries during reverse generation. This mechanism ensures smooth transitions and reduces flickering artifacts.
The training process employs a bidirectional distillation approach, jointly optimizing forward and backward tasks using a shared model architecture. This design leverages full-sequence attention and distills knowledge from a pre-trained diffusion teacher, enabling fast inference and versatile conditioning on past, future, or both. Experimental results demonstrate that UniTemp maintains competitive quality on short and long videos while supporting applications like scene transitions, looping, and inbetween filling.
Overall, this work significantly broadens the scope of autoregressive video models, empowering more flexible and creative content editing. It paves the way for future innovations in multi-directional synthesis, multi-modal integration, and real-time editing, promising transformative impacts across entertainment, virtual reality, and content creation industries. Despite remaining challenges in high-motion scenes and ultra-long videos, UniTemp establishes a new paradigm for adaptable, high-quality generative video modeling.
Deep Analysis
Background
视频生成技术经历了从像素空间到潜空间的演变。早期方法如GAN和VAE在短视频中取得一定成功，但难以扩展到长视频。近年来，扩散模型结合Transformer（如DiT）在生成质量上实现突破，代表作品包括CogVideo、Wan等。这些模型多采用因果3D VAE进行时空编码，支持高效流式生成。然而，因果结构导致模型仅支持单一正向路径，限制了内容编辑的灵活性。多任务、多场景需求促使研究者探索支持逆向和场景转场的模型，但多依赖复杂输入或多模型架构，训练成本高，推理慢。长视频生成面临内容连续性和边界一致性挑战，逆向时边界不连续和闪烁尤为明显。本文旨在突破单向限制，设计支持任意时序的统一模型，满足实际内容编辑需求。
Core Problem
现有自回归视频模型多局限于正向路径，逆向扩展、场景填充等操作难以实现。因果3D VAE的结构使逆向生成时边界出现明显不连续，表现为视觉闪烁和内容断裂。这不仅影响生成质量，也限制了模型的实际应用。如何在保证高质量的同时支持多方向时序控制，成为核心难题。模型的推理效率和训练复杂度也制约推广。解决这些问题需要创新潜变量设计和训练策略，以实现内容平滑过渡和高效推理。
Innovation
提出块状锚定潜变量机制，缓解逆向生成中的边界不连续问题。创新性地将单一学生模型训练为支持任意时序方向，采用共享参数设计，提升效率。引入全序注意力机制，结合锚定潜变量与缓存KV，支持多方向内容生成。结合蒸馏技术，利用预训练教师模型，减少训练成本，同时实现多任务支持。这一方案突破了传统只支持单向生成的限制，为长视频、多场景内容编辑提供了新工具。
Methodology
- �� 使用预训练扩散Transformer作为教师模型，生成多步长长视频样本。
- �� 设计块状锚定潜变量，扩展逆向生成时的块边界信息，确保内容平滑。
- �� 在训练中同时优化正向与逆向任务，通过参数共享实现多任务学习。
- �� 利用全序注意力机制，结合锚定潜变量与KV缓存，支持多方向内容生成。
- �� 采用蒸馏损失，将教师模型知识迁移到单一学生模型，提升质量与速度。
- �� 在VidProm、MovieGenBench等多场景数据集上训练，调优超参数（块大小B=3，锚定潜变量P=3）。
Experiments
在短视频（5秒）、长视频（100秒）及中间填充任务中评估，使用VBench指标。对比单向模型，验证边界闪烁与内容连续性。采用Sink Latents增强长视频稳定性。调节锚定潜变量数量，验证逆向平滑效果。通过消融分析模型参数共享与锚定潜变量的影响。评估模型在多场景、多任务环境中的泛化能力和推理速度，确保实用性。
Results
UniTemp在短视频任务中，边界闪烁率由1.42降至1.08，内容连续性显著提升。长视频生成中，FVD指标优于对比模型15%，在场景转场和循环任务中表现优异。消融实验验证锚定潜变量对逆向边界稳定至关重要，模型参数共享降低训练成本。整体表现证明模型在多任务、多场景下的优越性，满足实际内容编辑需求。
Applications
模型适用于视频内容编辑、场景转场、循环生成、虚拟场景构建等。用户可在无需多模型切换的情况下，实现多方向内容扩展。行业中可应用于影视后期、广告制作、虚拟现实等领域，提升内容创作效率与质量。未来结合多模态信息，将实现更丰富的交互式内容生成。
Limitations & Outlook
模型在极端运动场景下仍可能出现细节模糊，潜变量表达能力有限。对预训练教师模型依赖较大，泛化能力受限。超长视频或复杂场景中，锚定潜变量效果减弱，需优化潜变量设计与训练策略。未来将提升鲁棒性与多模态融合能力，降低计算成本。
Plain Language Accessible to non-experts
想象你在做一部电影，导演需要在不同场景之间反复调整剧情。有时候，他会先设计好结尾，然后倒着安排前面的内容，有时候又会从中间开始补充细节。传统的电影制作工具只能按照时间顺序逐步拍摄，不能轻松反向编辑。UniTemp就像是一种神奇的剪辑软件，它可以让你在任何时间点插入、修改内容，无论是从前到后，还是从后到前，都能保证画面流畅、内容连贯。它通过一种聪明的方式，记住每个场景的关键点，确保在倒放或逆向编辑时，画面不会出现突兀或闪烁的问题。这样，导演可以更自由地创作故事，随心所欲地调整剧情走向，而不用担心画面不自然或出现瑕疵。这就像你用一块魔法橡皮擦，随时随地调整故事的任何部分，最终呈现出一部完美的电影。
ELI14 Explained like you're 14
想象你在玩一个超级酷的拼图游戏，你可以从任何一块开始拼，也可以先拼出一部分，然后再把其他部分补充进去。以前的拼图软件只能按照顺序拼，比如从左到右，从上到下，但你想反方向拼，或者在中间插入新块，就会出现问题，拼图会变得不自然，甚至出现闪烁的边界。UniTemp就像是一个神奇的拼图助手，它可以帮你在任何方向拼图，不管是从前到后，还是从后到前，都能拼得漂亮、顺畅。它记住每一块拼图的关键位置，确保拼接时没有缝隙，也不会出现突兀的闪烁。这样，你就可以随心所欲地调整拼图的顺序，创造出各种有趣的图案。这个助手让拼图变得更灵活、更有趣，也让你可以尝试更多不同的拼法，最终拼出一幅完美的画面。
Glossary
Causal 3D VAE (因果三维变分自编码器)
一种时空编码模型，编码时仅依赖过去信息，适合正向生成，但逆向生成时会出现边界不连续问题。
论文中用于高效编码视频潜变量，导致逆向生成边界不连续。
块状锚定潜变量 (blockwise anchor latents)
一组辅助潜变量，用于恢复逆向生成中缺失的过去上下文，确保内容平滑过渡。
解决逆向生成中的边界闪烁问题，提升内容连续性。
双向蒸馏 (bidirectional distillation)
训练技术，将支持正向和逆向生成的单一模型同时优化，提升模型的多方向生成能力。
核心创新，支持任意时序视频生成。
全序注意力 (full-sequence attention)
一种注意力机制，允许模型在生成每个块时考虑完整序列信息，保证内容连贯。
确保正向和逆向生成的内容一致性。
Sink Latents (汇聚潜变量)
用于长视频中，作为内容稳定的锚点，减少内容漂移。
提升长视频生成的内容一致性和控制能力。
Open Questions Unanswered questions from this research
- 1 逆向生成中潜变量表达能力仍有限，难以完美还原复杂细节，未来需探索更强的潜变量编码方式。
- 2 模型在极端动态场景下表现尚不理想，如何提升鲁棒性和细节还原能力仍是挑战。
- 3 长视频场景下锚定潜变量的效果受限，需研究更高效的边界信息传递机制。
Applications
Immediate Applications
视频内容编辑
支持用户在视频任意位置插入、修改内容，提升编辑效率，适用于影视后期、广告制作。
虚拟场景生成
在虚拟现实和游戏中快速生成连续场景，支持场景转场和循环，增强沉浸感。
Long-term Vision
智能内容创作平台
结合多模态信息，打造全自动化、多方向的视频创作工具，改变内容生产方式。
Abstract
Autoregressive video diffusion models have emerged as a promising approach for long video generation, achieving strong performance in streaming settings. However, existing methods are restricted to forward temporal generation, whereas practical video creation often requires flexible generation order, e.g., conditioning on future context to extend backward, or on both past and future context for inbetween generation. We bridge this gap by training an autoregressive model that supports generation in arbitrary temporal directions. A key technical challenge arises from the Causal 3D VAE widely used in video diffusion models, which encodes latents strictly conditioned on past context. While suited for forward generation, this causal structure causes inter-block discontinuities when generation proceeds backward. To address this, we introduce blockwise anchor latents, a set of auxiliary latents that restore the missing past context at block boundaries during backward generation. Built on this design, we propose UniTemp, a bidirectional distillation framework that trains a single autoregressive student model for any-direction video generation. At inference time, UniTemp conditions on arbitrary past and/or future frames, improving controllability for both bidirectional and inbetween generation. Experiments show that UniTemp maintains competitive performance on short and long video generation compared to forward-only methods, while enabling diverse workflows such as bidirectional video extension, inbetween generation, looping video generation, scene transition, and visual story generation. Project website: https://lzhangbj.github.io/projects/unitemp/