Long Context Tuning for Video Generation
Long Context Tuning (LCT) extends pre-trained video diffusion models' context window, enabling scene-level multi-shot generation with high consistency.
Key Findings
Methodology
This work builds upon a pre-trained single-shot video diffusion transformer (DiT), expanding its attention from individual shots to entire scenes via full attention mechanisms. It introduces interleaved 3D Rotary Positional Embedding (RoPE) to distinguish shots, and employs an asynchronous noise strategy to unify conditioning and diffusion sampling. The model is fine-tuned to support both joint and auto-regressive multi-shot generation without additional parameters. Data is curated from scene-level videos, with scene descriptions generated by Gemini-1.5, enabling training on scene-level coherence. The approach allows the model to learn scene-level visual and dynamic consistency, producing multi-shot videos with emergent capabilities.
Key Results
- Post-LCT, the model can generate around 20 shots lasting 3 minutes, maintaining high visual and semantic coherence, outperforming original single-shot models.
- It demonstrates scene-level consistency, including stable character identities and environment details, with scores reaching 95.65 in background consistency and an average user rating of 3.79/5.
- Emerging abilities include compositional generation, interactive shot extension, and zero-shot scene synthesis, significantly advancing multi-shot video generation.
Significance
This method addresses the longstanding challenge of scene-level video synthesis, bridging the gap between single-shot realism and multi-shot narrative coherence. It enables more practical, scalable content creation for film, VR, and gaming industries, providing a foundation for complex, long-duration virtual scenes with consistent visual storytelling.
Technical Contribution
The core innovation lies in extending full attention to entire scenes with interleaved 3D RoPE, enabling the model to learn cross-shot dependencies. The asynchronous noise strategy allows flexible conditioning, while fine-tuning the model to support causal attention improves inference efficiency. This combination results in a scalable, end-to-end scene-level video diffusion framework that preserves pre-trained single-shot quality while enabling multi-shot coherence.
Novelty
This work is the first to incorporate full attention mechanisms with interleaved 3D positional encoding for scene-level video generation, moving beyond prior methods reliant on keyframes or external conditions. It introduces a parameter-efficient approach that enables multi-shot, multi-scene synthesis within a unified diffusion framework, representing a significant step forward in long-duration video synthesis.
Limitations
- The model's performance degrades with extremely long or highly complex scenes, mainly due to data limitations and computational constraints.
- High-quality multi-shot generation demands substantial computational resources, limiting real-time applications.
- Scene descriptions depend on accurate annotations; biased or incomplete data can affect coherence and fidelity.
Future Work
Future research will focus on optimizing attention mechanisms for longer scenes, incorporating multi-modal cues such as audio and text, and improving inference speed. Extending the approach to more complex narrative structures and diverse scene types will further enhance its practical utility.
AI Executive Summary
The creation of coherent, multi-shot videos remains a significant challenge in computer vision and content generation. While recent diffusion-based models can produce realistic single-shot videos, extending this capability to scene-level narratives with multiple shots has been elusive. Traditional approaches rely heavily on keyframes or external conditioning, which often lead to inconsistencies and limited flexibility.
This paper introduces Long Context Tuning (LCT), a novel training paradigm that significantly expands the context window of pre-trained single-shot video diffusion models. By leveraging full attention mechanisms across entire scenes, combined with interleaved 3D Rotary Positional Embedding (RoPE), the model can distinguish individual shots while maintaining their relative relationships. An asynchronous noise strategy further unifies conditioning inputs and diffusion sampling, enabling both joint and auto-regressive multi-shot generation without adding parameters.
The core technical innovation lies in adapting the attention mechanism to encompass all scene shots, allowing the model to learn scene-level visual and dynamic consistency directly from data. Fine-tuning the model to support causal attention enhances inference efficiency, making auto-regressive generation feasible with KV-cache. Extensive experiments demonstrate that the LCT-enhanced model can generate scenes with around 20 shots, lasting over 3 minutes, with high visual fidelity and semantic coherence. The model exhibits emergent capabilities such as compositional scene synthesis, interactive shot extension, and zero-shot scene assembly, opening new avenues for practical content creation.
This work marks a significant advance in long-duration video synthesis, bridging the gap between single-shot realism and multi-shot narrative coherence. It offers a scalable, end-to-end framework suitable for film production, virtual reality, and interactive media. Despite current limitations in handling extremely complex scenes and computational costs, future directions include optimizing attention efficiency, integrating multi-modal cues, and expanding narrative complexity. Overall, LCT paves the way for more intelligent, flexible, and realistic virtual scene generation, promising transformative impacts across entertainment and industry sectors.
Deep Analysis
Background
近年来，视频生成技术快速发展，代表性工作包括Stable Diffusion Video、Lumiere和Emu Video等。这些模型多基于GAN或扩散机制，能生成高质量短视频，但在长时间、多场景、多镜头的场景级生成方面仍存在瓶颈。随着扩散变换器（DiT）[40]的提出，视频生成质量得到显著提升，尤其是在多模态融合和时间建模方面取得突破。然而，现有模型多局限于单镜头或短片，难以实现复杂场景的连续、多镜头一致性，限制了其在影视、虚拟现实等行业的应用潜力。场景级视频生成面临的主要挑战包括：跨镜头的视觉和动态一致性、复杂叙事结构的建模，以及长时间跨度内的连贯性维护。
Core Problem
核心问题在于如何在保持单镜头高质量生成的基础上，扩展模型的上下文范围，实现多镜头、多场景的连贯性。传统方法多依赖关键帧或外部条件，难以捕捉长时间内的动态变化和视觉一致性，且参数扩展带来计算成本。如何在不增加参数的情况下，提升模型对场景信息的理解和生成能力，是当前的技术难点。此外，长场景生成中的信息保持和动态一致性也是亟待解决的问题。
Innovation
本研究的创新点包括：1）引入全注意力机制，覆盖整个场景中的所有镜头，提升跨镜头信息整合能力；2）设计交错的3D RoPE编码，为每个镜头赋予唯一位置标识，避免信息混淆；3）采用异步噪声策略，实现条件融合和多镜头联合或自回归生成；4）微调模型为因果注意力结构，提升推理效率。这些创新共同实现了端到端的场景级视频生成框架，保持预训练单镜头模型的高质量输出，同时支持多镜头场景的连贯合成。
Methodology
- �� 以预训练的单镜头视频扩散模型（DiT）为基础，扩展全注意力机制覆盖整个场景中的所有镜头；• 设计交错的3D RoPE编码，为每个镜头赋予唯一位置标识，区分不同镜头，保持相对关系；• 采用异步噪声策略，为每个镜头设置不同噪声水平，实现条件融合和多镜头联合生成；• 训练过程中，结合场景描述和镜头信息，微调模型以学习场景级连贯性；• 采集多源场景视频数据，利用场景边界检测和镜头切割，丰富训练样本；• 支持联合和自回归生成，满足不同应用需求。
Experiments
采用Gemini-1.5生成场景描述数据，训练模型在多个场景和镜头数据集上。评估指标包括视觉质量、场景一致性和用户主观评价。对比基线包括关键帧方法和外部条件模型。调节最大上下文窗口（最多9个镜头），进行不同噪声水平和注意力结构的消融实验，验证模型在场景连贯性和复杂能力上的提升。模型在生成20镜头、持续3分钟的场景中表现出色，背景和角色位置保持稳定，获得95.65的背景一致性得分和3.79/5的用户平均评分。
Results
LCT后模型能生成连续3分钟、20镜头的场景，视觉和语义高度一致，背景和角色位置保持稳定。定量指标显示背景一致性得分达95.65，用户评价平均3.79/5，优于传统方法。模型还展现出复合生成和场景扩展能力，支持交互式编辑，验证了其在影视制作中的潜力。
Applications
该技术适用于电影场景制作、虚拟现实内容生成和游戏动画等领域，满足对长时连续、多镜头场景的高质量需求。未来可结合多模态信息（如声音、文本）增强场景理解，推动虚拟环境的智能化和个性化定制。
Limitations & Outlook
模型在极长或复杂场景中仍存在一致性下降的问题，计算资源消耗大，推理速度需优化。场景描述依赖高质量标注，数据偏差可能影响效果。未来需提升模型效率、扩展多模态融合能力，以实现更广泛应用。
Plain Language Accessible to non-experts
想象你在准备一场大型派对。每个桌子上的装饰、灯光和音乐都需要协调一致，不能只考虑一个桌子，而忽略整体氛围。传统方法就像只为每个桌子单独布置，缺乏整体感。而这项新技术像是用一份全局的设计图，确保每个细节都符合整体风格。它还能根据不同区域的需要，灵活调整灯光和装饰，让整个派对看起来既统一又丰富。通过这种方式，派对的每个部分都能和谐统一，给宾客带来完美体验。这就像让视频中的每个镜头都像是派对的一部分，整体连贯又精彩纷呈。
ELI14 Explained like you're 14
想象你在拍一部电影，但不想每个镜头都单独拍，而是希望整个故事看起来连贯又自然。以前的方法就像是只拍一段，然后自己拼接，可能会出现角色突然变脸或者场景不连贯的问题。现在，这个新技术就像是用一台超级智能的摄像机，它可以记住所有镜头的细节，把每个镜头都连起来，让整个故事看起来像是真实发生的一样。它还能根据故事的需要，自动补充一些细节，比如角色重新出现或者场景变化，完全不用担心不连贯的问题。这样一来，制作电影变得更快、更自然，也更容易让观众相信故事是真的发生在他们面前的。
Abstract
Recent advances in video generation can produce realistic, minute-long single-shot videos with scalable diffusion transformers. However, real-world narrative videos require multi-shot scenes with visual and dynamic consistency across shots. In this work, we introduce Long Context Tuning (LCT), a training paradigm that expands the context window of pre-trained single-shot video diffusion models to learn scene-level consistency directly from data. Our method expands full attention mechanisms from individual shots to encompass all shots within a scene, incorporating interleaved 3D position embedding and an asynchronous noise strategy, enabling both joint and auto-regressive shot generation without additional parameters. Models with bidirectional attention after LCT can further be fine-tuned with context-causal attention, facilitating auto-regressive generation with efficient KV-cache. Experiments demonstrate single-shot models after LCT can produce coherent multi-shot scenes and exhibit emerging capabilities, including compositional generation and interactive shot extension, paving the way for more practical visual content creation. See https://guoyww.github.io/projects/long-context-video/ for more details.