SemanticGen: Video Generation in Semantic Space
SemanticGen uses a two-stage diffusion framework in semantic space, boosting long video generation speed and quality, with a 30% faster convergence and better long-term consistency.
Key Findings
Methodology
SemanticGen employs a dual diffusion process: first, generating compact semantic features in a high-level semantic space using a diffusion model conditioned on semantic encoders like Qwen-2.5-VL; second, mapping these features to VAE latents via a lightweight MLP, then refining in the latent space. Semantic compression reduces high-dimensional semantic features into Gaussian distributions, accelerating training and sampling. Full attention is applied in semantic space for global planning, while shifted window attention manages complexity in the latent space, enabling efficient long video synthesis. The approach integrates semantic encoding, diffusion modeling, and attention mechanisms to improve convergence speed and content coherence.
Key Results
- On the VBench benchmark, SemanticGen achieves a 97.79% subject consistency rate for short videos, matching or surpassing SOTA models like Wan2.1-T2V-14B. For long videos, ∆Mdrift drops to 3.58, outperforming SkyReels-V2 and LongLive, indicating superior long-term stability. Training convergence improves by approximately 30%, significantly reducing content drift over extended sequences.
- Ablation studies confirm that semantic encoder choice and compression are critical; models without semantic guidance show lower quality and higher drift. The method scales well to videos up to one minute, maintaining high visual fidelity and temporal coherence.
- Compared to baseline models trained without semantic guidance, SemanticGen demonstrates faster convergence, higher quality, and better long-term consistency, validating the effectiveness of semantic space modeling.
Significance
This work addresses the fundamental bottleneck in long video generation: computational complexity and content drift. By shifting the modeling focus to a high-level semantic space, it drastically reduces the number of tokens involved in attention mechanisms, enabling faster training and inference. The approach enhances content stability over long sequences, opening new possibilities for applications like virtual production, immersive media, and interactive entertainment. It also paves the way for more scalable and controllable generative models, bridging the gap between high-quality synthesis and practical deployment in industry.
Technical Contribution
SemanticGen introduces a novel paradigm of semantic space diffusion, combining semantic encoding, compression, and attention mechanisms. It innovatively applies full attention in semantic space and shifted window attention in latent space, balancing efficiency and quality. The framework leverages pre-trained semantic encoders and diffusion models, integrating them through a learned Gaussian semantic prior, which accelerates convergence and improves long-term coherence. This approach extends the applicability of diffusion models to large-scale, long-duration video synthesis, marking a significant step forward in generative modeling.
Novelty
This is the first work to perform global video planning entirely within a compressed semantic space using diffusion models, significantly reducing computational complexity for long videos. Unlike prior methods that model in high-dimensional latent spaces directly, SemanticGen’s semantic-guided approach achieves faster convergence and better content stability. The integration of semantic encoding, compression, and attention mechanisms presents a new paradigm for scalable, high-quality long video synthesis, setting a new benchmark in the field.
Limitations
- The method relies heavily on pre-trained semantic encoders, which may not perform equally well across all scene types, potentially limiting generalization in highly complex or novel scenarios.
- Semantic compression may lead to loss of fine details, affecting the fidelity of small or intricate visual elements.
- Despite improvements, some residual content drift and repetitive patterns can still occur in very long sequences, requiring further model refinement.
Future Work
Future research will explore multi-modal semantic integration, such as combining audio and text cues, to enhance content richness. Improving semantic encoder robustness and developing adaptive compression strategies could further boost detail preservation. Additionally, incorporating self-supervised learning and multi-scale attention may reduce drift and improve diversity. Extending the framework to real-time applications and interactive editing also presents promising directions.
AI Executive Summary
The rapid growth of virtual content demands high-quality, long-duration videos, yet existing generative models face significant challenges. Traditional diffusion-based approaches, while capable of producing detailed short videos, struggle with scalability and content stability over extended sequences. As the length of generated videos increases, computational costs grow quadratically, and content drift accumulates, degrading the coherence and realism of the output.
SemanticGen introduces a groundbreaking framework that addresses these issues by shifting the generative process into a high-level semantic space. This space captures the global structure and dynamics of the video, enabling the model to plan scenes holistically before adding finer details. The core innovation lies in a two-stage diffusion process: first, generating compact semantic features conditioned on text prompts, then mapping these features into the VAE latent space for detailed rendering. This approach leverages pre-trained semantic encoders like Qwen-2.5-VL and employs a lightweight MLP to compress semantic features into Gaussian distributions, greatly accelerating training and sampling.
The model employs full attention mechanisms in the semantic space to model global dependencies efficiently, while using shifted window attention in the latent space to handle local details without prohibitive computational costs. Experimental results demonstrate that SemanticGen achieves comparable or superior quality to state-of-the-art models on short videos, with a 97.79% subject consistency, and significantly outperforms baselines on long videos, reducing drift to a ∆Mdrift of 3.58. Notably, training convergence speeds up by approximately 30%, and the generated videos maintain long-term coherence, even up to one minute.
This work marks a significant advance in scalable, high-quality long video synthesis. By decoupling global planning from detail refinement, SemanticGen opens new horizons for applications in virtual production, entertainment, and immersive media. Despite some limitations related to semantic encoder dependence and residual drift, ongoing research aims to incorporate multi-modal cues, improve semantic compression, and develop adaptive attention mechanisms. Overall, SemanticGen paves the way for more efficient, stable, and controllable video generation systems, promising a new era of virtual content creation.
Deep Dive
Glossary
Diffusion Model (扩散模型)
一种生成模型，通过逐步添加和去除噪声，学习数据分布，生成高质量样本。In this paper,用于视频生成的核心算法。
用于在潜在空间或语义空间中逐步生成视频内容。
VAE (变分自编码器)
一种深度生成模型，通过编码和解码潜在空间实现数据压缩与重建。In this研究中，用于视频潜在表示。
作为视频生成的基础潜在空间。
语义编码器
预训练模型，用于提取视频的高层语义特征。支持语义空间生成。
如Qwen-2.5-VL，用于压缩视频的语义信息。
Shifted Window Attention (偏移窗口注意力)
一种注意力机制，通过在不同窗口间移动，减少长序列建模的计算复杂度。提升长视频建模效率。
在潜在空间中实现长序列的高效建模。
长视频漂移 (drift)
生成过程中内容逐渐偏离主题或结构的现象。影响长视频的连贯性。
通过语义空间规划减缓漂移。
Open Questions Unanswered questions from this research
- 1 如何进一步提升语义编码器在复杂场景中的表现，确保多样化内容的准确捕捉。
- 2 在保持高效率的同时，如何减少语义空间压缩带来的细节损失，提升细节还原能力。
- 3 长视频生成中的漂移问题仍未完全解决，未来需探索更有效的模型结构和训练策略。
Applications
Immediate Applications
虚拟主播与虚拟偶像
利用SemanticGen快速生成高质量、连贯的虚拟人物视频，支持实时交互和内容定制，提升虚拟娱乐体验。
影视特效制作
为电影和动画提供长时间连续的虚拟场景，节省大量后期制作时间，增强视觉表现力。
Long-term Vision
虚拟现实与沉浸式内容
实现大规模、真实感强的虚拟场景，推动虚拟世界的普及，改变娱乐和教育方式。
Abstract
State-of-the-art video generative models typically learn the distribution of video latents in the VAE space and map them to pixels using a VAE decoder. While this approach can generate high-quality videos, it suffers from slow convergence and is computationally expensive when generating long videos. In this paper, we introduce SemanticGen, a novel solution to address these limitations by generating videos in the semantic space. Our main insight is that, due to the inherent redundancy in videos, the generation process should begin in a compact, high-level semantic space for global planning, followed by the addition of high-frequency details, rather than directly modeling a vast set of low-level video tokens using bi-directional attention. SemanticGen adopts a two-stage generation process. In the first stage, a diffusion model generates compact semantic video features, which define the global layout of the video. In the second stage, another diffusion model generates VAE latents conditioned on these semantic features to produce the final output. We observe that generation in the semantic space leads to faster convergence compared to the VAE latent space. Our method is also effective and computationally efficient when extended to long video generation. Extensive experiments demonstrate that SemanticGen produces high-quality videos and outperforms state-of-the-art approaches and strong baselines.