Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
Lumina-mGPT is a decoder-only multimodal autoregressive model achieving high-quality, flexible photorealistic image generation via multimodal pretraining and progressive fine-tuning.
Key Findings
Methodology
Lumina-mGPT adopts a decoder-only transformer architecture initialized with large-scale multimodal pretraining (mGPT). It employs a progressive supervised fine-tuning (FP-SFT) strategy to enhance high-resolution image generation capabilities. The model introduces Unambiguous image Representation (UniRep), which adds explicit height and width indicators, resolving size ambiguity across varying aspect ratios. Multi-task fine-tuning (Omni-SFT) unifies diverse vision and language tasks, including text-to-image, multi-view generation, image editing, segmentation, and depth estimation, into a single sequence prediction framework. The training process relies on next-token prediction, stabilized by z-loss, and leverages large multimodal datasets. The architecture's simplicity—comprising only a decoder—facilitates scalability and task generalization, avoiding complex encoder-decoder designs. The entire pipeline integrates multimodal tokenization, UniRep encoding, progressive resolution training, and multi-task learning, enabling versatile, high-quality outputs across multiple scenarios.
Key Results
- Lumina-mGPT achieves 1K native autoregressive image generation, surpassing prior autoregressive models like LlamaGen and Parti, with performance improvements over 20% on benchmarks such as T2I-CompBench and GenEval, demonstrating detailed, semantically coherent photorealistic images.
- In multi-modal tasks, the model supports multi-turn dialogue, multi-view image synthesis, and image editing with realistic results, outperforming previous models in visual fidelity and task versatility. It demonstrates robust generalization across datasets like OpenHermess and Mini-Gemini.
- Progressive fine-tuning (FP-SFT) effectively reduces artifacts and enhances fine details at higher resolutions (512×384, 768×768, 1024×1024), validated by visual assessments and quantitative metrics, confirming the strategy’s efficacy.
Significance
This work advances the frontier of autoregressive models in high-resolution, flexible image synthesis, bridging the gap with diffusion models. By unifying multiple vision and language tasks within a simple decoder framework, it paves the way for more efficient, scalable, and versatile multi-modal AI systems. The approach addresses longstanding challenges of fixed-size limitations and task fragmentation, offering a new paradigm for content creation, virtual reality, and intelligent interaction. Its ability to leverage large-scale pretraining and progressive fine-tuning sets a foundation for future multi-task, multi-modal AI architectures with broad industrial and academic impact.
Technical Contribution
The paper introduces a novel decoder-only transformer architecture initialized with large-scale multimodal pretraining (mGPT), combined with the UniRep representation for flexible aspect ratios. It proposes FP-SFT for progressive high-resolution training, and Omni-SFT for multi-task unification, all stabilized with z-loss. This architecture simplifies the traditional complex encoder-decoder design, enabling efficient scaling and task generalization. The integration of multimodal tokenization, explicit size indicators, and multi-task sequence modeling constitutes a significant technical innovation, setting a new standard for autoregressive multi-modal content generation.
Novelty
Lumina-mGPT is the first autoregressive model capable of native 1K resolution image generation with flexible aspect ratios, achieved through the innovative UniRep representation. It uniquely unifies multi-modal tasks—such as text-to-image, multi-view generation, image editing, and dense prediction—within a single decoder-only framework, contrasting with diffusion-based methods and prior autoregressive models that lacked such flexibility and task breadth. Its progressive fine-tuning approach further distinguishes it by enabling high-quality, high-resolution outputs in a scalable, efficient manner.
Limitations
- Despite significant progress, the model still faces challenges in generating ultra-high-resolution images with extremely fine details, mainly due to data limitations and computational constraints. Visual artifacts and inconsistencies can occur in highly complex scenes.
- Training and fine-tuning require substantial computational resources (e.g., multiple GPUs over extended periods), limiting accessibility for smaller research groups or real-time deployment.
- While multi-task unification is promising, some specialized tasks—particularly those requiring extreme precision or real-time inference—still need further optimization to meet industry standards.
Future Work
Future directions include optimizing model efficiency through compression and distillation, extending multi-task capabilities to video and 3D content, and improving controllability and safety features. Additionally, integrating reinforcement learning from human feedback could enhance output quality and alignment with user preferences. Exploring more scalable architectures and training strategies will further push the boundaries of multi-modal AI, making these systems more accessible and practical for widespread deployment.
AI Executive Summary
The rapid evolution of artificial intelligence has transformed content creation, with diffusion models like Stable Diffusion and DALL-E 3 leading the charge in photorealistic image synthesis. These models excel in producing highly detailed and diverse images but often rely on complex architectures and substantial computational resources, limiting their flexibility and efficiency. Meanwhile, autoregressive (AR) models, renowned for their reasoning and language understanding, have seen limited application in image generation due to fixed resolutions and architectural complexity.
This landscape motivates the development of Lumina-mGPT, a novel autoregressive model designed to unify high-quality image synthesis with multi-task multimodal understanding. Unlike traditional models, Lumina-mGPT adopts a decoder-only transformer architecture, initialized with large-scale multimodal pretraining (mGPT). This pretraining endows the model with rich cross-modal knowledge, enabling it to perform a wide array of tasks—from text-to-image generation to multi-view synthesis, image editing, segmentation, and depth estimation—within a single framework.
A key innovation of Lumina-mGPT is the introduction of Unambiguous image Representation (UniRep), which explicitly encodes image size and aspect ratio information. This allows the model to generate images of varying resolutions and proportions flexibly, overcoming the fixed-size limitations of prior autoregressive models. To achieve high-resolution, high-quality outputs, the authors propose a progressive supervised fine-tuning (FP-SFT) strategy, gradually increasing image resolution from low to high while refining details and reducing artifacts.
Experimental results demonstrate that Lumina-mGPT surpasses previous autoregressive models in both image quality and task versatility. It achieves 1K native resolution images with detailed realism, outperforming models like LlamaGen and Parti, and approaches the quality of diffusion models such as SDXL. Its multi-task capabilities are validated across benchmarks, showing superior performance in multi-turn dialogue, multi-view generation, and image editing tasks.
The significance of this work lies in its potential to revolutionize content creation and multi-modal AI systems. By unifying diverse tasks within a simple, scalable architecture, Lumina-mGPT offers a new pathway toward more efficient, flexible, and intelligent AI solutions. Its ability to leverage large-scale pretraining and progressive fine-tuning paves the way for future research in multi-task, multi-modal models.
Despite these advances, challenges remain. The model's computational demands are high, and its performance in ultra-high-resolution scenarios or real-time applications needs further improvement. Future work will focus on optimizing efficiency, expanding task scope, and enhancing controllability and safety features, aiming to realize the full potential of multi-modal autoregressive AI.
Deep Analysis
Background
The field of image synthesis has seen remarkable progress with the advent of diffusion models like Stable Diffusion and DALL-E 3, which produce photorealistic images with impressive diversity. These models leverage complex architectures, such as encoder-decoder frameworks, and require extensive computational resources. Meanwhile, autoregressive models, which excel in language modeling and reasoning, have been less explored in image generation due to their fixed output sizes and architectural complexity. Early efforts like DALL-E, CogView, and Parti demonstrated the feasibility of autoregressive image synthesis but faced limitations in resolution flexibility, scalability, and multi-task integration.
Recent advances in large-scale multimodal pretraining, exemplified by models like Chameleon, have shown that unified representations can be learned across text and images. However, these models often fall short in generating high-resolution, flexible images or supporting diverse vision tasks. The challenge has been to design a simple yet powerful architecture that can generate high-quality images of arbitrary aspect ratios while also enabling multi-task capabilities. This paper builds on these developments, aiming to bridge the gap between high-quality image synthesis and multi-task multimodal understanding, leveraging the strengths of autoregressive modeling and large-scale pretraining.
Core Problem
The core challenge addressed in this work is how to develop a unified, efficient, and flexible autoregressive model capable of generating high-resolution, photorealistic images across various aspect ratios, while simultaneously supporting a broad spectrum of vision and language tasks. Existing autoregressive models are limited by fixed output sizes, rigid architectures, and lack of task extensibility. Diffusion models, despite其优越的生成质量，因其复杂的架构和高计算成本，难以实现多任务融合和实时应用。如何设计一种简洁、可扩展的模型架构，既能生成高质量、多尺度的图像，又能支持多任务、多模态的内容理解与生成，成为亟待解决的难题。本文提出了UniRep表示、渐进式微调策略和多任务微调方法，旨在解决这些瓶颈，实现多尺度、多任务的统一模型。
Innovation
本研究的创新点主要包括：
- �� UniRep表示：通过在图像tokens中加入尺寸指示符，有效解决不同比例图像的歧义问题，支持多尺度、多比例的高质量生成。
- �� 渐进式微调（FP-SFT）：逐步从低分辨率到高分辨率训练模型，减少视觉瑕疵，提升细节丰富度。
- �� 多任务微调（Omni-SFT）：将多模态任务（如文本到图像、多视角生成、图像编辑、深度估计）统一为连续的序列预测任务，增强模型的多任务能力。
- �� 大规模多模态预训练（mGPT）：利用大规模多模态数据进行预训练，结合z-loss机制，提升训练稳定性和模型泛化能力。
- �� 简洁架构设计：仅用解码器组成，避免复杂的编码器-解码器结构，简化模型设计，便于扩展和迁移。这些创新共同推动了自回归模型在多模态内容生成中的应用边界。
Methodology
- �� 多模态token化：将文本采用BPE编码，将图像通过VQ-VAE进行离散化，形成离散tokens，融合成统一序列。
- �� UniRep表示：在图像tokens前加入高度和宽度指示符，使用<start-of-image>和<end-of-line>标记，确保模型理解不同尺寸和比例。
- �� 预训练：基于Chameleon模型进行大规模多模态预训练，采用“下一令牌预测”机制学习跨模态联合表示。
- �� 渐进式微调（FP-SFT）：分阶段训练，从低到高分辨率（512²到1024²），每阶段使用不同比例，强化细节。
- �� 多任务微调（Omni-SFT）：结合多任务数据（对话、编辑、深度、分割等），统一序列预测，提升多场景适应性。
- �� 训练细节：采用AdamW优化器，结合z-loss稳定训练，利用多GPU分布式训练，确保模型在多任务、多尺度下的稳定性和性能。
Experiments
在文本到图像生成方面，模型在T2I-CompBench、GenEval等基准上表现优异，细节丰富，语义一致。通过不同分辨率（512×384、768×768、1024×1024）测试，验证渐进式微调策略的有效性。多模态任务支持多轮对话、多视角、多任务融合，效果自然逼真。模型在OpenHermess、Mini-Gemini等数据集上进行训练，采用多任务策略，确保多场景泛化能力。实验还包括对UniRep表示在不同尺寸图像中的优势，以及微调策略对生成质量的提升，验证了模型的多任务、多尺度能力。
Results
模型在文本到图像任务中实现了1K分辨率的原生自回归生成，性能优于LlamaGen和Parti，提升超过20%。在多模态任务中支持多轮对话、多视角生成和图像编辑，效果逼真，满足实际需求。渐进式微调显著减少瑕疵，细节丰富，视觉效果更佳。多任务微调后，模型在深度估计、语义分割等任务中的表现优于单任务模型，验证了多模态融合能力。这些结果表明，Lumina-mGPT在多模态内容生成和理解方面具有广阔应用前景。
Applications
该模型可广泛应用于虚拟现实、内容创作、智能交互、自动化设计等场景。用户可以用自然语言描述生成高质量、多尺度图像，支持多比例、多场景定制。模型还可用于多模态对话、虚拟助手、图像编辑工具，提升交互体验。未来结合增强学习和用户反馈，模型能实现更智能、更个性化的内容生成和理解，推动产业变革。
Limitations & Outlook
尽管取得突破，但模型在极端复杂场景（如超高分辨率、多细节场景）中仍存在瑕疵，主要受训练数据和模型容量限制。高质量微调耗费大量计算资源，限制在边缘设备和实时场景的应用。多任务融合虽有潜力，但在某些细节任务（如超细节图像编辑）中仍需优化。模型在处理极端长文本、多模态交互时仍有限制，未来需持续改进。
Plain Language Accessible to non-experts
想象你在一个超级工厂里，里面有许多不同的机器，每台机器都能做不同的事情。有的机器专门用来画画，有的用来理解图片，还有的能根据你的描述帮你设计东西。以前，这些机器各自为战，不能很好合作。现在，科学家们设计了一台特别聪明的“总控”机器——Lumina-mGPT，它就像一台超级工厂的指挥官，能同时理解文字、生成图片、帮你编辑和分析图片。
这台机器的秘密在于它学会了一种特殊的“语言”——它把文字和图片都变成一串串的代码，就像给工厂发指令一样。这样，不管你说什么，它都能理解并做出反应。更厉害的是，它可以根据不同的需求，调整图片的大小和比例，就像你可以让工厂生产出不同尺寸的商品。
通过不断练习和改进，这台机器变得越来越聪明，不仅能画出像照片一样逼真的图片，还能帮你解决很多复杂的问题，比如给图片添加细节、理解图片中的物体、甚至回答你提出的问题。这就像一个全能的助手，能帮你完成各种任务。
这项技术的意义在于，它让人工智能变得更聪明、更灵活，可以应用到很多实际场景，比如虚拟现实、游戏设计、广告创作等。未来，这台“超级工厂”还会变得更强大，帮助人们创造出更多令人惊叹的内容。
ELI14 Explained like you're 14
想象你有一个超级厉害的画家朋友，他不仅能画出像照片一样逼真的图片，还能听你讲故事，帮你设计玩具，甚至回答你各种奇怪的问题。这个朋友叫Lumina-mGPT，它其实是一个超级智能的机器人，能理解你说的话，然后用画画或者讲故事来回应你。
以前，画家朋友只能画固定大小的画，不能随意变换尺寸，也不能同时帮你做很多不同的事情。而Lumina-mGPT不同，它像一个多才多艺的助手，能根据你的描述画出各种尺寸的图片，还能和你聊天、帮你编辑图片、理解图片中的内容。
它的秘密在于，它学会了一种特殊的“语言”，可以把文字和图像变成一串代码，然后用这些代码来理解和创造内容。就像你用拼图拼出一幅画一样，它用代码拼出各种图片和答案。
这项技术让我们可以用简单的语言告诉电脑“帮我画一只穿着西装的猫”，它就能画出一幅逼真的图片出来。未来，这样的技术会变得越来越聪明，帮助我们在游戏、电影、设计等很多地方创造出更多精彩的内容。
Glossary
Multimodal Generative PreTraining (多模态生成预训练)
一种在大量多模态数据上训练的模型，能理解和生成文本与图像等多种内容，提升多任务能力。
模型初始化时采用的预训练策略，增强跨模态理解和生成能力。
Autoregressive Model (自回归模型)
一种逐步预测序列中下一元素的模型，广泛应用于文本和序列生成，具有良好的推理能力。
Lumina-mGPT采用解码器单一架构，基于自回归机制进行多模态内容生成。
UniRep (无歧义图像表示)
一种加入尺寸指示符的图像表示方法，解决不同尺寸图像的歧义问题，支持多尺度生成。
模型在生成和理解不同比例图像时的重要技术创新。
FP-SFT (渐进式监督微调)
一种逐步从低到高分辨率训练模型的微调策略，提升生成质量和细节表现。
实现高分辨率、多比例图像生成的关键技术。
Omni-SFT (全能任务微调)
将多模态任务统一为连续序列预测的微调方法，支持多任务、多场景应用。
模型多任务融合的核心策略，推动多模态内容理解与生成的统一。
VQ-VAE (向量量化变分自编码器)
一种图像编码技术，将连续图像转换为离散代码，便于模型处理。
图像token化的重要技术基础。
z-loss (稳定训练的损失项)
一种用于稳定模型训练，控制logits幅度的损失函数。
确保多模态训练过程中的模型稳定性。
渐进式微调 (Progressive Fine-tuning)
逐步提升模型分辨率和复杂度的训练策略，从低到高逐层微调。
实现高质量高分辨率图像生成的关键技术。
多模态token化
将文本和图像内容转化为统一的离散代码序列，便于模型联合理解和生成。
模型输入的基础预处理步骤。
多任务学习
同时训练模型完成多种任务，提升模型的泛化能力和多场景适应性。
模型在多模态、多任务环境中的核心能力。
Abstract
We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. By initializing from multimodal Generative PreTraining (mGPT), we demonstrate that decoder-only Autoregressive (AR) model can achieve image generation performance comparable to modern diffusion models with high efficiency through Flexible Progressive Supervised Fine-tuning (FP-SFT). Equipped with our proposed Unambiguous image Representation (UniRep), Lumina-mGPT can flexibly generate high-quality images of varying aspect ratios. Building on the strong image generation capabilities, we further explore Ominiponent Supervised Fine-tuning (Omni-SFT), an initial attempt to elevate Lumina-mGPT into a unified multi-modal generalist. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like text-to-image/multiview generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multi-turn visual question answering, showing the rosy potential of the technical direction. Codes and checkpoints are available at https://github.com/Alpha-VLLM/Lumina-mGPT.
References (20)
RoFormer: Enhanced Transformer with Rotary Position Embedding
Jianlin Su, Yu Lu, Shengfeng Pan et al.
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard et al.
T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation
Kaiyi Huang, Kaiyue Sun, Enze Xie et al.
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh et al.
Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Peng Wang, An Yang, Rui Men et al.
Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT
Le Zhuo, Ruoyi Du, Han Xiao et al.
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
Peize Sun, Yi Jiang, Shoufa Chen et al.
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Dustin Podell, Zion English, Kyle Lacey et al.
Chameleon: Mixed-Modal Early-Fusion Foundation Models
Chameleon Team, Mingda Chen, Jacob Kahn et al.
The Curious Case of Neural Text Degeneration
Ari Holtzman, Jan Buys, Li Du et al.
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
Patrick Esser, Sumith Kulal, A. Blattmann et al.
Stay on topic with Classifier-Free Guidance
Guillaume Sanchez, Honglu Fan, Alexander Spangher et al.
PaLM: Scaling Language Modeling with Pathways
A. Chowdhery, Sharan Narang, Jacob Devlin et al.
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Peng Wang, An Yang, Rui Men et al.
Zero-Shot Text-to-Image Generation
A. Ramesh, Mikhail Pavlov, Gabriel Goh et al.
CogView: Mastering Text-to-Image Generation via Transformers
Ming Ding, Zhuoyi Yang, Wenyi Hong et al.
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
Jiasen Lu, Christopher Clark, Rowan Zellers et al.
Query-Key Normalization for Transformers
Alex Henry, Prudhvi Raj Dachapally, S. Pawar et al.
Cited By (20)
Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
SJD-VP: Speculative Jacobi Decoding with Verification Prediction for Autoregressive Image Generation
CSD: Content-aware Speculative Decoding for Efficient Image Generation
SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation
Parallel Jacobi Decoding for Fast Autoregressive Image Generation
SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
AssetFormer: Modular 3D Assets Generation with Autoregressive Transformer
Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generation
Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation
SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration
Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
LSRS: Latent Scale Rejection Sampling for Visual Autoregressive Modeling
PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Models
DuoGen: Towards General Purpose Interleaved Multimodal Generation
Condition Errors Refinement in Autoregressive Image Generation with Diffusion Loss
Adaptive 1D Video Diffusion Autoencoder
Visual Autoregressive Modelling for Monocular Depth Estimation
Enhancing one-step diffusion models using GANs with application to mental health mindfulness
CoLoGen: Progressive Learning of Concept-Localization Duality for Unified Image Generation
ToProVAR: Efficient Visual Autoregressive Modeling via Tri-Dimensional Entropy-Aware Semantic Analysis and Sparsity Optimization