VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models

The evolution of generative models from GANs and VAEs to transformers and diffusion models has significantly advanced visual synthesis. Early video generation relied on GANs like VGAN and MoCoGAN, which struggled with temporal consistency. The advent of transformers (e.g., VideoGPT, CogVideo) improved sequence modeling. Recently, diffusion models such as Video Diffusion Models (VDMs) have demonstrated remarkable capabilities in image synthesis, inspiring their extension to video. Notable works include VideoVAE, Imagen Video, and Make-a-Video, which utilize pixel and latent space strategies. Despite progress, high-quality video datasets remain scarce, with WebVid-10M providing diversity but limited resolution (~320p). Collecting and filtering high-quality videos is costly, limiting research and industrial deployment.

Current models heavily depend on large-scale, high-quality video datasets, which are expensive and difficult to acquire. WebVid-10M, while diverse, offers low resolution and inconsistent quality, impeding the training of high-fidelity models. This bottleneck restricts the development of applications requiring detailed, realistic videos. Moreover, models trained solely on low-quality data often suffer from poor visual quality and unstable motion. The challenge lies in achieving high-quality, natural videos without relying on costly datasets, necessitating innovative strategies to leverage existing resources effectively.

The paper introduces a data-level disentanglement approach: 1) analyzing the coupling between spatial and temporal modules in SD-based models, 2) leveraging full training on low-quality videos to learn motion, 3) selectively fine-tuning only the spatial modules with high-quality images to enhance appearance and concept richness. This strategy circumvents the need for high-quality videos by decoupling appearance and motion learning. Additionally, synthetic images with complex concepts are used to boost the model’s conceptual diversity. The approach combines theoretical analysis with practical pipeline design, enabling high-quality video synthesis from limited data, a significant departure from traditional reliance on expensive datasets.

• �� Start with a pre-trained Stable Diffusion model, extend it with spatial and temporal modules based on 3D-UNet architecture. • Analyze the connection strength between modules under different training strategies, observing that full training results in stronger coupling, which affects the model's robustness. • Use WebVid-10M to train the full model, capturing motion information. • Fine-tune the spatial module with high-quality images (JDB) using LORA or direct methods, while freezing temporal modules. • Employ synthetic images with complex concepts to further enhance concept composition. • Design a pipeline: full training on low-quality videos, then only spatial fine-tuning with high-quality images, followed by concept augmentation. • Validate through ablation studies, user preferences, and quantitative metrics.

• �� Datasets include WebVid-10M for low-quality videos and JDB plus LAION for high-quality images. • Evaluation metrics encompass visual quality, motion consistency, text-video alignment, using EvalCrafter and user studies. • Training involves 270K iterations on 32 A100 GPUs, with a learning rate of 5e-5. • Fine-tuning uses LORA on 8 GPUs for 30K iterations, resolution adjusted to 512×512. • Ablation studies compare full training, partial training, and fine-tuning strategies, confirming that only spatial fine-tuning yields optimal results. • User surveys show 64% preference for the proposed method over baseline models.

• �� The model achieves a visual quality score of 63.28, matching or surpassing models trained on high-quality videos. • User preference surveys favor the proposed approach in visual quality (56%) and motion stability (64%). • Synthetic concept images significantly improve the diversity and conceptual richness of generated videos, validated through qualitative and quantitative metrics. • The approach effectively balances visual fidelity and motion consistency, demonstrating robustness across various prompts and scenarios.

• �� Enables cost-effective high-quality video creation for entertainment, advertising, and virtual reality without expensive datasets. • Suitable for rapid prototyping, concept visualization, and personalized content generation. • Future integration with multi-modal inputs (audio, text) can expand its utility in immersive media, gaming, and remote communication.

• �� Performance may degrade in highly dynamic or cluttered scenes due to limited motion details in low-quality videos. • Fine-tuning sensitivity requires careful hyperparameter tuning; generalization across architectures needs further validation. • Current results focus on specific datasets; broader testing is necessary to confirm robustness in real-world applications.