Goku: Flow Based Video Generative Foundation Models
Goku employs rectified flow Transformer for joint image-video generation, achieving top-tier performance with 0.76 on GenEval and 84.85 on VBench.
Key Findings
Methodology
Goku integrates a 3D joint VAE with rectified flow (RF) to encode images and videos into a shared latent space, enabling unified modeling. It employs a full-attention Transformer architecture with 2B and 8B parameters, trained through multi-stage strategies including text-to-image pretraining, joint multimodal learning, and modality-specific fine-tuning. Data curation involves large-scale datasets with multimodal captions generated via multimodal large language models (MLLMs). Infrastructure optimizations include 3D parallelism, activation checkpointing, and fault-tolerant mechanisms, ensuring efficient training. RF models are trained by predicting velocity fields through linear interpolation between noise and data, accelerating convergence compared to diffusion models.
Key Results
- Goku surpasses prior models with 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image tasks. In text-to-video, it scores 84.85 on VBench, outperforming commercial counterparts. On ImageNet-1K, RF achieves lower FID after 400k steps versus DDPM's 1 million, demonstrating faster convergence. Qualitative samples show high coherence and detail, validating the model's robustness across modalities and complex scenes.
- Across benchmarks, Goku demonstrates superior visual quality, semantic alignment, and temporal consistency. Its multi-scale training and large dataset enable strong generalization. Ablation studies confirm the effectiveness of rectified flow in speeding up training and improving sample fidelity. The model's architecture supports diverse applications, from content creation to virtual reality.
Significance
This work advances multimodal generative modeling by introducing a unified rectified flow Transformer framework, addressing longstanding issues of slow convergence and inconsistent quality in joint image-video synthesis. Its scalable infrastructure and multi-stage training set a new standard for industrial-grade content generation. The ability to generate high-fidelity, temporally coherent media at scale opens new horizons for entertainment, advertising, and immersive experiences. Moreover, the integration of large datasets and multimodal captioning techniques enhances semantic understanding, making the models more aligned with human perception. Overall, Goku bridges the gap between research and practical deployment, pushing the frontier of AI-driven multimedia creation.
Technical Contribution
Key innovations include adapting rectified flow as a generative mechanism for multimodal synthesis, integrating full attention to model complex dependencies, and employing multi-stage training with large-scale data. Infrastructure improvements such as 3D parallelism, activation checkpointing, and fault tolerance enable training of models with billions of parameters efficiently. The shared latent space via joint VAE reduces computational complexity, while the flow formulation accelerates convergence. These contributions collectively set a new benchmark in speed, quality, and scalability for multimodal generative models.
Novelty
This is the first application of rectified flow in joint image and video generation, combined with a full-attention Transformer architecture for multimodal tasks. The multi-stage training pipeline, leveraging large-scale datasets and multimodal captioning, distinguishes Goku from prior diffusion or GAN-based models. Its ability to accelerate convergence and improve coherence across modalities marks a significant leap forward, filling a critical gap in the field.
Limitations
- Despite high quality, the model struggles with highly dynamic scenes requiring precise motion continuity, especially in long videos. Training costs remain substantial, demanding extensive GPU resources, limiting accessibility. The accuracy of multimodal captions depends heavily on language models, which may introduce semantic errors. Future work should focus on reducing computational overhead, improving temporal consistency, and enhancing semantic alignment.
Future Work
Future directions include optimizing model compression and inference speed, integrating more advanced language understanding for captions, and extending capabilities to longer, more complex videos. Exploring reinforcement learning and user feedback for personalized content generation is also promising. Additionally, developing more efficient training algorithms and hardware-aware architectures will facilitate broader deployment in industry settings.
AI Executive Summary
Goku represents a groundbreaking step in multimodal content generation, unifying image and video synthesis through a novel rectified flow Transformer architecture. Traditional approaches often treat images and videos separately, leading to inefficiencies and inconsistent quality. Goku’s core innovation lies in combining a shared latent space via a 3D joint VAE with a rectified flow mechanism, which transforms noise into high-fidelity visual content through a linear interpolation process. This approach simplifies training, accelerates convergence, and enhances sample coherence. The Transformer backbone, based on GenTron, employs full attention to model complex spatial-temporal dependencies across modalities, supported by advanced infrastructure including 3D parallelism, activation checkpointing, and fault-tolerant mechanisms, enabling large-scale training with billions of parameters.
Extensive experiments demonstrate Goku’s superiority across multiple benchmarks. For text-to-image generation, it achieves 0.76 on GenEval and 83.65 on DPG-Bench, outperforming existing models. In text-to-video tasks, it scores 84.85 on VBench, surpassing commercial solutions and setting new industry standards. The training process leverages massive datasets—36 million video-text pairs and 160 million image-text pairs—curated through sophisticated filtering, captioning, and balancing pipelines involving multimodal large language models. These datasets ensure rich semantic alignment and diversity.
The practical implications are profound: Goku can generate high-quality, temporally coherent media for virtual reality, entertainment, advertising, and beyond. Its scalable infrastructure and multi-stage training pipeline provide a blueprint for future large-scale multimodal models. Despite current limitations in handling extremely complex scenes and high computational costs, ongoing research aims to optimize efficiency and extend capabilities. Overall, Goku paves the way for more intelligent, versatile, and efficient multimedia content creation, bridging cutting-edge research and real-world applications.
Deep Dive
Abstract
This paper introduces Goku, a state-of-the-art family of joint image-and-video generation models leveraging rectified flow Transformers to achieve industry-leading performance. We detail the foundational elements enabling high-quality visual generation, including the data curation pipeline, model architecture design, flow formulation, and advanced infrastructure for efficient and robust large-scale training. The Goku models demonstrate superior performance in both qualitative and quantitative evaluations, setting new benchmarks across major tasks. Specifically, Goku achieves 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for text-to-video tasks. We believe that this work provides valuable insights and practical advancements for the research community in developing joint image-and-video generation models.