ModelScope Text-to-Video Technical Report
ModelScopeT2V uses diffusion with spatio-temporal blocks, 1.7B parameters, achieving superior text-to-video synthesis with high temporal coherence.
Key Findings
Methodology
ModelScopeT2V employs VQGAN encoder, pre-trained CLIP text encoder, and denoising UNet, integrating novel spatio-temporal blocks to model dependencies across frames. The model operates in latent space with diffusion sampling, conditioned on text via cross-attention. It supports variable frame numbers during training and inference, with 0.5 billion parameters dedicated to temporal modeling within a total of 1.7 billion. The architecture extends Stable Diffusion by incorporating spatial-temporal convolution and attention modules, capturing motion and scene dynamics effectively. Multi-modal training on image-text and video-text datasets enhances semantic understanding, leading to high-quality video generation.
Key Results
- On MSR-VTT, ModelScopeT2V achieves FID-vid of 11.09 and FVD of 550, outperforming baselines like Make-A-Video (FID 13.17). CLIPSIM scores 0.293, indicating strong semantic alignment. Qualitative results show superior motion coherence and detail restoration, especially in complex scenes. Ablation studies confirm the effectiveness of the spatio-temporal blocks and multi-modal training, with parameters optimized for capturing long-range dependencies. The model demonstrates robustness across diverse scenarios, producing realistic, continuous videos.
- The integration of spatial and temporal modules contributes approximately 39% of the total UNet parameters, significantly improving temporal dependency modeling. The multi-frame training strategy, combining image-text and video-text datasets, enhances semantic richness and reduces motion artifacts. The model's ability to generate high-fidelity videos with variable frame counts marks a notable advancement, supported by extensive quantitative and qualitative evaluations. Despite the large parameter count, training efficiency is maintained through pretraining and freezing parts of the network.
- Ablation experiments show that both temporal convolution and attention contribute roughly 15-20% to performance gains, with their combination yielding optimal results. The model excels in capturing motion continuity and scene consistency, validated by metrics and visual comparisons. Limitations include challenges in extremely complex scenes and long videos, where motion blur and content deviations occur due to data limitations and computational constraints.
Significance
This work pioneers open-source diffusion-based text-to-video generation, bridging the gap between static image synthesis and dynamic video creation. It introduces a scalable, flexible architecture capable of producing high-quality, semantically aligned videos, fostering research and industry applications in virtual content, entertainment, and AR/VR. The multi-modal training approach and spatio-temporal modeling set new standards, enabling more natural and diverse video synthesis. The model’s open availability accelerates community-driven innovation, making advanced video generation accessible for broader use cases.
Technical Contribution
The paper introduces a novel spatio-temporal block combining convolution and attention mechanisms in the latent space, significantly improving dynamic modeling. It extends the Stable Diffusion framework to handle variable frame counts, with 0.5 billion parameters dedicated to temporal dependencies. The multi-modal training pipeline effectively leverages image-text and video-text datasets, enhancing semantic comprehension. The architecture supports flexible inference, enabling high-quality, continuous video synthesis. These innovations collectively push the boundaries of diffusion models in video generation, offering new avenues for research and practical deployment.
Novelty
This is the first work to incorporate space-time separated convolution and attention modules within a diffusion-based latent space for video synthesis. The multi-modal training strategy, combining image-text and video-text data, enhances semantic understanding and motion continuity. The support for variable frame numbers during training and inference is a key innovation, providing flexibility absent in prior models. These contributions establish a new paradigm in diffusion-based video generation, setting a foundation for future scalable, high-fidelity models.
Limitations
- Despite improvements, the model struggles with highly complex scenes and very long videos, often exhibiting motion blur and scene inconsistencies, mainly due to limited training data diversity and computational resources. The large parameter size (~1.7B) results in high computational costs, restricting real-time applications. Further, the model's ability to generate highly detailed scenes remains limited by current architecture and training data, requiring future research to address these issues for broader deployment.
Future Work
Future directions include integrating multi-condition inputs such as depth, lighting, and style to enhance realism. Expanding training datasets with more diverse and longer videos will improve scene complexity handling. Model efficiency can be optimized via pruning or distillation to enable real-time applications. Additionally, exploring reinforcement learning and self-supervised approaches could further improve content fidelity and temporal coherence, paving the way for scalable, high-fidelity long-video synthesis.
AI Executive Summary
ModelScopeT2V represents a significant advancement in diffusion-based text-to-video synthesis. Built upon the Stable Diffusion architecture, it introduces innovative spatio-temporal blocks that effectively model motion and scene dynamics. By leveraging a combination of convolutional and attention mechanisms in the latent space, the model captures complex dependencies across frames, supporting variable frame counts during training and inference. With a total of 1.7 billion parameters—of which 0.5 billion focus on temporal modeling—it demonstrates superior performance on benchmarks such as MSR-VTT, achieving an FID-vid of 11.09 and FVD of 550, surpassing existing models like Make-A-Video.
Quantitative evaluations show that ModelScopeT2V produces videos with high visual fidelity and semantic alignment, validated by metrics and qualitative comparisons. The model excels in generating smooth, realistic motion and detailed scenes, especially in complex scenarios. Its multi-modal training strategy, combining image-text and video-text datasets, enhances semantic understanding, enabling more accurate and diverse video content creation. The open-source release fosters community engagement, encouraging further innovation.
Looking ahead, the authors plan to incorporate multi-condition inputs, optimize efficiency for real-time applications, and extend the model’s capacity for longer, more complex videos. These developments promise to further bridge the gap between static image synthesis and dynamic video generation, opening new horizons for AI-driven content creation across entertainment, virtual reality, and beyond.
Deep Dive
Abstract
This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.