Emu3: Next-Token Prediction is All You Need
Emu3 employs a single Transformer trained solely with next-token prediction, surpassing SDXL and LLaVA-1.6 in multimodal tasks.
Key Findings
Methodology
Emu3 adopts a pure next-token prediction framework, utilizing discrete tokens for images, videos, and text. A vision tokenizer based on SBER-MoVQGAN encodes visual data into discrete tokens, which are then integrated with textual tokens using special markers such as [BOS], [EOS], [SOV], and [SOT]. The model architecture extends Llama-2 by enlarging the embedding layer to accommodate visual tokens, employing Rotary Positional Embedding (RoPE) and SwiGLU activation functions. During training, the model maximizes the cross-entropy loss across multimodal sequences, with additional weighting for vision tokens. The training process involves two stages: initial training on text and image data, followed by incorporating video data with extended context length up to 131072 tokens. Fine-tuning with quality preference optimization (QPO) and direct preference optimization (DPO) further enhances generation quality and alignment with human preferences.
Key Results
- In image generation benchmarks such as MSCOCO-30K, GenEval, T2I-CompBench, and DPG-Bench, Emu3 outperforms autoregressive diffusion models and matches or exceeds the performance of DALL-E 3 and SDXL. For example, on GenEval, Emu3 scored 85.4, surpassing Chameleon’s 78.2, indicating superior text-to-image alignment and content fidelity.
- In vision understanding tasks, Emu3 achieved an average score above 70 across 12 benchmarks, including SEEDBench, RealWorldQA, and OCRBench, demonstrating robust multimodal reasoning capabilities. It excels particularly in OCR and visual question answering, validating its effectiveness in complex perception tasks.
- For video generation, Emu3 utilizes next-token prediction to generate high-fidelity videos, with a VBench score of 80.6, outperforming many open-source models. It can extend videos temporally and predict future frames, showcasing strong temporal modeling and scene consistency.
Significance
This work fundamentally challenges the prevailing paradigm that multimodal AI requires complex architectures like diffusion or multi-component systems. By demonstrating that a simple next-token prediction approach, combined with discrete multimodal tokens, can achieve state-of-the-art results, it opens new avenues for scalable, efficient, and unified multimodal models. The approach simplifies the design, reduces computational costs, and enhances the potential for large-scale training, making multimodal AI more accessible and versatile. Its success suggests that future models could leverage this paradigm to achieve more general, flexible, and human-like understanding and generation capabilities, impacting fields from content creation to human-computer interaction.
Technical Contribution
The key technical innovations include: • Development of a high-quality vision tokenizer based on SBER-MoVQGAN, enabling discrete representation of images and videos. • Extension of the Transformer architecture to handle multimodal sequences with extremely large context lengths (up to 131072 tokens). • Integration of multimodal data through special tokens and unified sequence modeling, simplifying the architecture. • Implementation of a training pipeline that combines pretraining, quality fine-tuning, and preference alignment (DPO), optimizing both generation quality and human preference adherence. • Demonstration that pure next-token prediction can effectively handle complex multimodal tasks, including high-resolution image synthesis, video generation, and multimodal understanding, without diffusion or explicit multi-component systems.
Novelty
This research is the first to successfully implement a fully autoregressive, next-token prediction-based multimodal model that unifies vision, language, and video tasks within a single architecture. Unlike prior works that relied on separate modules or diffusion mechanisms, Emu3 leverages discrete tokens and a simplified, scalable design. Its ability to generate and understand across modalities with a single model represents a significant departure from existing paradigms, marking a new direction in multimodal AI research.
Limitations
- Despite its strengths, Emu3 faces challenges in extremely complex scenes involving multiple interacting objects or very high-resolution videos, where details may be lost or inconsistencies arise due to limited training data diversity and model capacity.
- High-quality video generation remains computationally intensive, requiring substantial hardware resources, which limits real-time deployment and widespread adoption.
- The discrete visual tokens, while efficient, may not fully capture fine-grained details in high-resolution or highly textured scenes, necessitating further improvements in tokenization strategies.
- The model's performance in out-of-distribution scenarios or novel domains still needs validation, and its generalization capabilities require further exploration.
Future Work
Future research will focus on enhancing the visual tokenizer's expressiveness, reducing computational costs, and improving the model's ability to handle high-resolution, multi-object scenes. Incorporating reinforcement learning and human preference feedback could further align outputs with user expectations. Expanding multimodal datasets with diverse and complex content will be crucial for improving robustness and generalization. Additionally, exploring more efficient training paradigms and model compression techniques will facilitate deployment in real-world applications, bringing this promising approach closer to practical use.
AI Executive Summary
Emu3 marks a transformative step in multimodal artificial intelligence, demonstrating that a single, unified Transformer model trained solely with next-token prediction can outperform specialized architectures based on diffusion or modular design. This breakthrough hinges on the innovative discretization of visual data into tokens, enabling the model to process images, videos, and text within a common autoregressive framework.
The core idea is to encode images and videos into discrete tokens using a high-quality vision tokenizer based on SBER-MoVQGAN. These tokens are then integrated with textual tokens through specially designed sequence markers, forming a comprehensive multimodal input. The Transformer architecture, inspired by Llama-2, is expanded to support extremely long contexts (up to 131072 tokens), allowing the model to handle complex, multi-step tasks across modalities.
Training involves a two-stage process: initial pretraining on large-scale text and image data, followed by incorporating video data with longer sequences. To further refine performance, the authors employ quality fine-tuning (QFT) and direct preference optimization (DPO), aligning the model outputs with human aesthetic and semantic preferences. The result is a versatile model capable of high-fidelity image synthesis, detailed visual understanding, and dynamic video generation.
Experimental results demonstrate Emu3’s competitive edge: surpassing state-of-the-art models like SDXL and LLaVA-1.6 in vision tasks, matching DALL-E 3 in image quality, and achieving top scores in video benchmarks such as VBench. The model’s ability to generate coherent, high-resolution videos and extend sequences temporally showcases its potential for real-world applications in entertainment, virtual reality, and content creation.
This work challenges the conventional reliance on diffusion models and complex multi-component systems, proposing a simpler yet more powerful paradigm. By focusing solely on tokens and next-word prediction, Emu3 simplifies the architecture, reduces training costs, and enhances scalability. Its success paves the way for future research into unified, scalable, and general-purpose multimodal AI, with broad implications for both academia and industry.
Despite these advances, challenges remain, including handling extremely complex scenes, reducing computational demands, and improving tokenization fidelity. The authors outline future directions such as enhancing visual tokenizer expressiveness, expanding datasets, and integrating reinforcement learning to better align with human preferences. Overall, Emu3 exemplifies a promising new direction in the quest for artificial general intelligence, emphasizing simplicity, scalability, and versatility in multimodal learning.
Deep Analysis
Background
多模态人工智能的研究经历了从单一模态模型到融合多模态信息的逐步演变。早期工作如ImageNet图像识别和BERT文本理解奠定了基础。随着对视觉和语言结合需求的增长，出现了CLIP、ALIGN等对比学习模型，实现了文本与图像的对齐。随后，生成模型如DALL-E、Stable Diffusion引入扩散机制，极大提升了图像生成质量，但架构复杂、训练成本高。视频理解与生成成为新热点，采用GAN和扩散模型，但在效率和细节保持方面仍有不足。近年来，Transformer架构的普及推动多模态模型快速发展，但多依赖复杂的多组件设计。Emu3的出现，试图通过离散化tokens和纯下一词预测，简化架构，提升效率，推动多模态AI走向更高的通用性。
Core Problem
现有多模态模型多依赖复杂架构，如扩散模型或多模态融合机制，导致训练成本高、模型难以扩展。在处理高分辨率、多对象、多场景的复杂内容时，表现出细节不足和一致性差的问题。尤其是在视频生成领域，逐帧或扩散机制的高计算成本限制了实时应用。如何在保持高性能的同时，简化模型设计、提升可扩展性，成为核心难题。解决方案需兼顾生成质量、推理速度和模型复杂度，推动多模态AI的实用化。
Innovation
Emu3的创新点包括：1）提出纯粹基于下一词预测的多模态架构，摒弃复杂扩散或多组件设计，简化模型；2）开发高效视觉tokenizer（基于SBER-MoVQGAN），实现图像和视频的离散编码，提升表达能力；3）引入多模态序列融合策略，利用特殊标记整合文本、图像、视频信息，支持端到端训练；4）扩展Transformer架构，支持超大上下文（131072 tokens），实现长序列建模；5）结合质量微调和偏好优化技术，提升生成质量和偏好一致性。这些创新共同推动了多模态模型的简洁性和性能提升。
Methodology
- �� 数据准备：采集多模态数据（文本、图像、视频），通过多阶段过滤（分辨率、美学、运动、文本检测）筛选高质量样本。• 视觉tokenizer：基于SBER-MoVQGAN训练，将图像和视频编码为离散tokens，支持多分辨率和时间维度压缩。• 模型架构：借鉴Llama-2，扩展嵌入层以支持视觉tokens，采用Rotary Positional Embedding和SwiGLU激活。• 输入构建：融合文本、视觉tokens和特殊标记（如[BOS]、[EOS]、[SOV]、[SOT]），形成多模态序列。• 训练目标：利用交叉熵损失，调节vision token的权重，进行端到端训练。• 预训练策略：分两个阶段，基础训练（无视频）和引入长视频序列，最大上下文长度131072。• 后续微调：通过QFT增强生成质量，结合DPO优化模型偏好。
Experiments
采用MSCOCO-30K、GenEval、T2I-CompBench、DPG-Bench等公开数据集进行评估。对比多种扩散模型和自回归方法，采用FID、IS、人工评价等指标。训练中调节超参数（学习率5×10^-5，批次大小，训练轮数），进行消融验证模型设计的有效性。视频生成部分，使用VBench进行多维度评估，包括内容一致性、动作自然度和场景丰富性。模型还在未来帧预测和视频延伸任务中表现优异，验证其时序建模能力。
Results
在图像生成方面，Emu3在MSCOCO-30K、GenEval等基准上得分均优于多种自回归扩散模型，尤其在生成内容丰富、细节逼真方面表现突出。视觉理解任务中，模型在12个公开基准中平均得分超过70，特别是在OCR和问答任务中表现优异。视频生成方面，模型在VBench中获得80.6分，优于大部分开源模型，支持长序列生成和未来帧预测，验证其时间建模能力。偏好优化后，模型生成质量和人类偏好一致性进一步提升，显示出良好的应用潜力。
Applications
该模型适用于自动内容生成（图片、视频、动画）、虚拟现实、增强现实、智能交互等场景。无需复杂架构，易于扩展，适合大规模部署。未来结合强化学习和偏好优化，有望实现个性化、多样化的内容创作，推动娱乐、教育、设计等行业的变革。
Limitations & Outlook
模型在极端复杂场景（如多对象交互、超高分辨率视频）下仍存在细节缺失和一致性问题，主要源于训练数据多样性不足和模型容量限制。高质量视频生成依赖大量计算资源，成本较高，限制了实时应用。离散视觉tokens的表达能力有限，可能在高细节场景表现不佳。未来需优化编码策略，提升泛化能力，降低训练成本，以实现更广泛的实际应用。
Plain Language Accessible to non-experts
想象你在一个大型工厂里，工厂里有很多不同的机器，每台机器负责不同的任务，比如制造零件、装配产品、检测质量。过去，每台机器都需要专门设计，操作复杂，成本高。而现在，Emu3就像是一个超级智能的工厂管理系统，只用一种简单的指令（下一词预测）就能控制所有机器，让它们协同工作，生产出各种不同的产品。它把图片、视频和文字都变成一种特殊的“代码”，就像用一种统一的语言交流。这样一来，工厂的设计变得更简单，生产也更快、更灵活。这个系统还能预测未来的场景，就像提前知道明天的天气一样，帮助我们更好地规划工作。它的出现，让我们可以用更少的资源，创造出更丰富、更逼真的内容，比如动画、虚拟场景，甚至虚拟人物。
ELI14 Explained like you're 14
想象你在玩一个超级厉害的游戏，你可以让游戏里的角色做任何事情，比如跳舞、跑步、说话。以前，要让角色做这些动作，可能要用很多复杂的指令或者动画软件。而现在，有了Emu3，就像你有一个聪明的朋友，只需要告诉它一句话，比如“一个穿着蓝色T恤的孩子在草地上笑”，它就能帮你画出这个场景，甚至还能让场景动起来，像电影一样。这个朋友特别聪明，因为它学会了用一种特别的语言，把图片、视频和文字都变成一样的“秘密代码”。它用这个“秘密代码”理解世界，然后用同样的“语言”帮你创造内容。最酷的是，它还能预测未来会发生什么，比如让动画角色跑得更远，或者场景变得更热闹。这个技术就像拥有一个会看会说、还能预知未来的超级机器人，让我们的娱乐和创作变得更有趣、更方便！
Abstract
While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.
References (20)
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al.
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Dustin Podell, Zion English, Kyle Lacey et al.
GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
Dhruba Ghosh, H. Hajishirzi, Ludwig Schmidt
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Xi-Wei Hu, Rui Wang, Yixiao Fang et al.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Pan Lu, Swaroop Mishra, Tony Xia et al.
Visual Instruction Tuning
Haotian Liu, Chunyuan Li, Qingyang Wu et al.
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Deyao Zhu, Jun Chen, Xiaoqian Shen et al.
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Wenliang Dai, Junnan Li, Dongxu Li et al.
Evaluating Object Hallucination in Large Vision-Language Models
Yifan Li, Yifan Du, Kun Zhou et al.
Improving Image Generation with Better Captions
James Betker, Gabriel Goh, Li Jing et al.
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan et al.
Flamingo: a Visual Language Model for Few-Shot Learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc et al.
Hierarchical Text-Conditional Image Generation with CLIP Latents
A. Ramesh, Prafulla Dhariwal, Alex Nichol et al.
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
Ahmed Masry, Do Xuan Long, J. Tan et al.
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Junnan Li, Dongxu Li, Caiming Xiong et al.
High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, A. Blattmann, Dominik Lorenz et al.
CogView: Mastering Text-to-Image Generation via Transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong et al.
InfographicVQA
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito et al.
RoFormer: Enhanced Transformer with Rotary Position Embedding
Jianlin Su, Yu Lu, Shengfeng Pan et al.
Cited By (20)
Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning
SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation
Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks
Obliviate: Erasing Concepts from Autoregressive Image Generation Models
Bridging Video Understanding and Generation in a Unified Framework
Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation
Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation
Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models
Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression
SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation
MemLearner: Learning to Query Context memory for Video World Models
UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation
WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression
MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts
C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Space
Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process
Transferability Between Understanding and Generation in Unified Multimodal Models
Unified Audio Intelligence Without Regressing on Text Intelligence
Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation