Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute
Proposes an efficient zero-shot subject-driven video generation framework using only 1% compute resources.
Key Findings
Methodology
The study introduces a zero-shot subject-driven video generation framework by decomposing the problem into identity injection and motion awareness tasks, learned from subject-image pairs and arbitrary videos, respectively. A stochastic task-switching strategy optimizes these tasks, avoiding test-time per-subject tuning and large-scale subject-video pairs.
Key Results
- Using the CogVideoX-5B model, trained with 200K subject-image pairs and 4,000 arbitrary videos in 288 A100 GPU hours, achieving only 1% compute compared to previous zero-shot baselines.
- Maintains competitive subject fidelity and motion quality compared to baselines like VACE and Phantom.
- Experiments show that identity and motion updates concentrate on disjoint parameter subspaces, reducing interference.
Significance
This study significantly reduces the computational cost of subject-driven video generation, making personalized video generation more accessible. By reducing dependency on large-scale subject-video pairs, it lowers the entry barrier in this field, promoting widespread application of video personalization and customization.
Technical Contribution
By decomposing identity and motion learning tasks, the study proposes a novel optimization strategy that avoids the mutual interference seen in traditional methods. It provides an effective path for zero-shot video generation without subject-video pairs.
Novelty
This is the first to decompose subject-driven video generation into identity injection and motion awareness tasks, using a stochastic task-switching strategy to significantly reduce compute requirements.
Limitations
- May encounter motion consistency issues in dynamic scenes, especially when the subject is small in the initial frame.
- Further research is needed to improve model generalization in low-data environments.
Future Work
Future work could explore applying this method in more complex scenarios and further reducing computational costs. Researchers could also attempt to apply this method to other types of generation tasks.
AI Executive Summary
Subject-driven video generation is a challenging task, with traditional methods requiring extensive computational resources and data. Existing methods often rely on large-scale subject-video pairs, leading to high computational costs. This study proposes a novel zero-shot framework by decomposing the problem into identity injection and motion awareness tasks, significantly reducing compute requirements.
The method uses the CogVideoX-5B model, combined with 200K subject-image pairs and 4,000 arbitrary videos, trained in 288 A100 GPU hours. Through a stochastic task-switching strategy, the model maintains high-quality subject fidelity and motion consistency without using subject-video pairs.
This research offers new possibilities for video personalization and customization, lowering the entry barrier in this field. However, challenges remain in addressing motion consistency issues in dynamic scenes and improving generalization in low-data environments.
Deep Analysis
Background
Video generation technology has made significant progress in recent years, particularly in text-to-video synthesis and video customization. Traditional subject-driven video generation methods require separate tuning for each subject, leading to high computational costs. Zero-shot methods, while avoiding per-subject tuning, rely on large-scale subject-video pairs, limiting their application.
Core Problem
The core problem of subject-driven video generation is how to generate videos that preserve the identity of the subject while allowing variations in scene, motion, and context. Existing methods' high computational cost and data requirements limit their usability.
Innovation
The core innovation of this study is decomposing the subject-driven video generation problem into identity injection and motion awareness tasks. By using a stochastic task-switching strategy, the model can learn subject identity features and motion patterns without using subject-video pairs.
Methodology
- �� Use CogVideoX-5B model for experiments
- �� Learn identity features from 200K subject-image pairs
- �� Learn motion patterns from 4,000 arbitrary videos
- �� Optimize tasks using stochastic task-switching strategy
- �� Employ LoRA technique to reduce computational cost
Experiments
Experiments use the CogVideoX-5B model, combined with 200K subject-image pairs and 4,000 arbitrary videos. Through a stochastic task-switching strategy, the model is trained in 288 A100 GPU hours. The experiments evaluate the model's performance in subject fidelity and motion consistency.
Results
The experimental results show that the method maintains high-quality subject fidelity and motion consistency while achieving only 1% compute compared to previous zero-shot baselines. It remains competitive with baselines like VACE and Phantom.
Applications
This method can be used for personalized video generation and customized content creation, suitable for scenarios requiring efficient generation of diverse video content, such as advertising, entertainment, and education.
Limitations & Outlook
The method may encounter motion consistency issues in dynamic scenes, especially when the subject is small in the initial frame. Additionally, further research is needed to improve the model's generalization in low-data environments.
Plain Language Accessible to non-experts
Imagine a factory with two production lines, one responsible for the appearance of the product and the other for its motion functions. The factory's goal is to produce products that are both aesthetically pleasing and functionally complete. Our research is like this factory, handling the product's appearance and function through two separate lines. This way, even without a lot of raw materials, the factory can efficiently produce high-quality products.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to create a character. This character not only has to look cool but also move smoothly in the game. Our research is like a super-smart game editor that helps you quickly create such a character without spending a lot of time adjusting every detail. Isn't that awesome?
Glossary
Zero-Shot
Refers to the model's ability to apply to new tasks without specific training data.
In this study, zero-shot methods are used for subject-driven video generation.
Identity Injection
Learning and maintaining the subject's identity features from subject-image pairs.
Used to ensure consistency of the subject's identity in generated videos.
Motion Awareness
Learning and maintaining motion patterns from arbitrary videos.
Used to ensure natural motion effects in generated videos.
Stochastic Task Switching
Randomly switching optimization targets between different tasks during training.
Used to optimize between identity injection and motion awareness.
LoRA (Low-Rank Adaptation)
A technique used to reduce computational costs of models.
Used in this study to reduce computational resources during training.
Open Questions Unanswered questions from this research
- 1 How to improve motion consistency in dynamic scenes? Current methods may struggle with complex motions, requiring new solutions.
- 2 How to improve model generalization in low-data environments? Current methods perform poorly with insufficient data.
Applications
Immediate Applications
Personalized Video Generation
Advertising and entertainment industries can use this method to quickly generate personalized video content, enhancing user engagement.
Long-term Vision
Educational Content Creation
The education industry can use this method to generate diverse teaching videos, enhancing the learning experience.
Abstract
Subject-driven video generation (SDV-Gen) aims to produce videos of a specific subject by adapting a pretrained video model, enabling personalized and application-driven content creation. To achieve this goal, per-subject tuning methods require approximately 200 A100 GPU hours to generate a customized video, whereas zero-shot methods avoid per-subject tuning but typically rely on millions of subject-video pairs for the supervision, incurring massive network fine-tuning costs (10K-200K A100 GPU hours). We propose a data- and compute-efficient zero-shot SDV-Gen framework that avoids test-time per-subject tuning and the use of large-scale subject-video pairs. Our key idea decomposes SDV-Gen into (i) identity injection learned from subject-image pairs and (ii) motion-awareness preservation maintained by a small set of arbitrary videos. We optimize the two tasks with stochastic switching, using random reference-frame sampling and image-token dropout to prevent trivial first-frame copying. Our gradient analysis shows that the two objectives rapidly evolve toward nearly orthogonal update subspaces, explaining the stable optimization. Using CogVideoX-5B, we adapt a single model with 200K subject-image pairs and 4,000 arbitrary videos in 288 A100 GPU hours. This yields about 1% of compute compared to prior zero-shot baselines (i.e., 0.4% of VACE and 2.8% of Phantom) while using no subject-video pairs, yet remaining competitive in subject fidelity and motion quality. We show that the same recipe transfers to Wan 2.1-1.3B and Wan 2.2-5B.