Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Proposes Tune-A-Video, a one-shot tuning method for text-to-video generation using pretrained diffusion models, achieving high content consistency with minimal fine-tuning.
Key Findings
Methodology
This work extends pretrained 2D text-to-image diffusion models (like Stable Diffusion) into the spatio-temporal domain by inflating convolutional layers and incorporating sparse 3D attention. A key innovation is updating only the attention projection matrices during fine-tuning, which preserves the original model knowledge while adapting to video data. During inference, DDIM inversion extracts structural information from a source video, guiding the sampling process for temporal coherence. The model employs a sparse causal attention mechanism across frames to reduce computational complexity from quadratic to linear in the number of frames, enabling efficient long-video generation. The approach integrates structure guidance and inverse modeling to produce smooth, content-consistent videos from a single text-video pair.
Key Results
- On 42 DAVIS videos, the method achieves a frame consistency score of 92.4%, surpassing CogVideo's 90.64%. The CLIP-based textual alignment score reaches 27.58, outperforming baselines. Fine-tuning takes only 10 minutes, with sampling in 1 minute, drastically reducing computational costs. Ablation studies confirm that the sparse spatio-temporal attention and structure guidance significantly improve motion continuity and content fidelity.
- Across applications like object editing, background change, and style transfer, the model demonstrates strong content preservation and controllability. It can incorporate personalized models like DreamBooth, enabling customized video synthesis. Quantitative comparisons show superior performance in both content consistency and textual alignment, validating the approach’s effectiveness.
- The combination of sparse attention and structure-guided inversion maintains high quality in long videos, though challenges remain in multi-object scenes and complex motions. Future work will focus on enhancing multi-object handling, improving motion details, and scaling to higher resolutions.
Significance
This method addresses the high cost and data dependency of traditional video generation by enabling single-sample tuning of powerful pretrained image diffusion models. It offers a practical, scalable solution for personalized, high-quality text-driven video synthesis, broadening the accessibility of multimodal content creation. The approach significantly reduces training time and computational resources, making real-time, controllable video generation feasible. Its innovative use of sparse spatio-temporal attention and structure guidance paves the way for future research in efficient, long-duration video synthesis, with applications spanning entertainment, virtual reality, and content creation industries.
Technical Contribution
The paper introduces a novel framework that adapts 2D diffusion models to the spatio-temporal domain via inflation and sparse attention. It proposes a parameter-efficient fine-tuning strategy that updates only attention projection matrices, preserving pre-trained knowledge. The integration of DDIM inversion for structure guidance enhances motion continuity. The sparse causal attention mechanism reduces complexity from quadratic to linear with respect to frame count, enabling scalable long-video synthesis. These innovations collectively push the boundary of efficient, controllable, and high-fidelity text-to-video generation.
Novelty
This is the first work to perform one-shot tuning of pretrained image diffusion models for video generation, avoiding large-scale video dataset training. It introduces a sparse spatio-temporal attention mechanism and combines structure-guided inversion, enabling high-quality, temporally coherent videos from a single sample. Unlike prior methods relying on extensive video data, this approach leverages existing image models, significantly reducing costs and complexity while maintaining content fidelity and motion continuity.
Limitations
- The method struggles with scenes involving multiple objects and occlusions, often leading to content mixing or motion artifacts due to the sparse attention mechanism’s limited capacity to capture complex interactions.
- Fine-tuning still requires around 10 minutes per video, which may be prohibitive for real-time applications or high-resolution long videos.
- Current approach is less effective in dynamic backgrounds and highly complex motions, necessitating further improvements in multi-object and multi-action modeling.
Future Work
Future directions include developing more sophisticated attention mechanisms to better handle multi-object interactions, extending the framework to higher resolutions and longer videos, and integrating multi-modal cues such as depth or audio for richer content synthesis. Additionally, exploring unsupervised or weakly supervised methods could further reduce reliance on annotated data, making the technology more accessible for broader applications.
AI Executive Summary
The rapid growth of multimodal content creation has spurred interest in text-driven video synthesis. Traditional approaches rely heavily on large-scale video datasets for training, which are costly and limit personalization. To address these challenges, this paper introduces Tune-A-Video, a novel framework that leverages pretrained image diffusion models for efficient, one-shot tuning to generate high-quality, temporally coherent videos from a single text-video pair.
At its core, the method extends 2D diffusion models into the spatio-temporal domain by inflating convolutional layers and incorporating sparse causal attention across frames. A key innovation is updating only the attention projection matrices during fine-tuning, which preserves the original model’s knowledge while adapting to new video content. During inference, DDIM inversion extracts structural information from the source video, guiding the sampling process to produce smooth, content-consistent motion.
Extensive experiments on 42 DAVIS videos demonstrate that Tune-A-Video surpasses state-of-the-art baselines like CogVideo in both frame consistency (92.4% vs. 90.64%) and textual alignment (CLIP score 27.58). The entire process takes only about 10 minutes for tuning and 1 minute for sampling, making it highly efficient. The framework supports a variety of applications, including object editing, background modification, style transfer, and personalized content creation, validating its versatility.
This approach significantly reduces the reliance on large-scale video datasets, lowering computational costs and enabling rapid, customizable video synthesis. Its innovative use of sparse attention and structure-guided inversion opens new avenues for scalable, high-fidelity, controllable video generation. Future work will focus on enhancing multi-object handling, scaling to higher resolutions, and integrating multi-modal cues, promising a transformative impact on multimedia content creation and virtual reality industries.
Deep Dive
Abstract
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator. Despite their promising results, such paradigm is computationally expensive. In this work, we propose a new T2V generation setting$\unicode{x2014}$One-Shot Video Tuning, where only one text-video pair is presented. Our model is built on state-of-the-art T2I diffusion models pre-trained on massive image data. We make two key observations: 1) T2I models can generate still images that represent verb terms; 2) extending T2I models to generate multiple images concurrently exhibits surprisingly good content consistency. To further learn continuous motion, we introduce Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy. At inference, we employ DDIM inversion to provide structure guidance for sampling. Extensive qualitative and numerical experiments demonstrate the remarkable ability of our method across various applications.