Muse: Text-To-Image Generation via Masked Generative Transformers
Muse: a masked Transformer for text-to-image generation with FID 6.06, enabling fast, high-fidelity synthesis.
Key Findings
Methodology
Muse employs a masked modeling approach in discrete token space, integrating pre-trained T5-XXL text embeddings. It encodes images via VQGAN into discrete tokens at 256×256 and 512×512 resolutions. The core architecture includes a base masked Transformer predicting randomly masked image tokens, conditioned on unmasked tokens and text embeddings, and a super-resolution Transformer translating low-res tokens into high-res ones. Training involves variable masking rates and cross-entropy loss, while inference uses parallel decoding to predict multiple tokens simultaneously, dramatically reducing sampling steps. The model parameters range from 632M to 3B, trained on the Imagen dataset, achieving state-of-the-art FID scores of 6.06 on CC3M and 7.88 on zero-shot COCO, with a CLIP score of 0.32.
Key Results
- The 632M model attains an FID of 6.06 on CC3M, surpassing prior methods; the 3B model achieves 7.88 on COCO with CLIP 0.32. Inference is over 10× faster than diffusion or autoregressive models, with comparable or superior image quality. It effectively captures objects, spatial relations, pose, and counts, supporting zero-shot editing tasks like inpainting, outpainting, and style transfer.
- Quantitative evaluations show Muse outperforms existing models in both quality and efficiency. It demonstrates strong understanding of language nuances, multi-object relationships, and style variations, generating diverse, high-fidelity images aligned with complex prompts.
- The model’s ability to perform zero-shot editing without fine-tuning, combined with its speed and accuracy, makes it highly practical for real-world applications such as content creation, virtual environments, and assistive design tools.
Significance
Muse marks a significant advance in text-to-image synthesis by balancing high quality with unprecedented inference speed. Its discrete token and masked modeling paradigm addresses the bottleneck of slow diffusion sampling and autoregressive decoding, enabling real-time applications. The integration of large language models enhances semantic understanding, allowing nuanced control over generated content. This work paves the way for scalable, versatile AI content generation, impacting industries from entertainment to e-commerce, and fostering new forms of human-AI creative collaboration.
Technical Contribution
The paper introduces a novel masked Transformer architecture operating on discrete image tokens, conditioned on rich language embeddings from a frozen large language model. It innovates with variable masking rates, parallel decoding, and multi-scale generation via cascaded models, achieving high efficiency without sacrificing quality. The approach combines the strengths of discrete token modeling, pre-trained language understanding, and efficient inference, setting new benchmarks in speed and fidelity. It also demonstrates zero-shot editing capabilities, broadening the scope of generative applications.
Novelty
This is the first work to unify masked Transformer modeling with discrete image tokens conditioned on large language models for text-to-image synthesis. Unlike pixel-based diffusion or autoregressive models, it employs parallel decoding and variable masking, resulting in over tenfold speed improvements. Its multi-scale, zero-shot editing features further distinguish it from prior methods, offering a versatile and efficient generative framework.
Limitations
- Despite high performance, the model struggles with very long or complex prompts, often misrepresenting details or failing to capture nuanced relationships, especially with high object counts.
- Large models (3B parameters) require substantial computational resources, limiting deployment on edge devices or in real-time scenarios without further optimization.
- Generation quality for highly stylized or rare scenes remains imperfect, indicating a need for more diverse training data and multimodal integration to enhance robustness.
Future Work
Future directions include optimizing decoding strategies like distillation or ODE-based sampling to further accelerate inference. Enhancing understanding of complex language structures and multi-object interactions is also a priority. Extending the framework to video and 3D content generation, integrating multimodal pretraining, and improving style control are promising avenues to broaden Muse’s applicability and performance.
AI Executive Summary
Muse introduces a novel masked Transformer architecture operating on discrete image tokens, conditioned on rich language embeddings from a frozen large language model (T5-XXL). This approach leverages a combination of VQGAN-based tokenization and a two-stage generation process—base masked Transformer predicting masked tokens and a super-resolution Transformer refining low-res outputs into high-resolution images. The core innovation lies in using variable masking rates and parallel decoding, which drastically reduces sampling steps from hundreds to fewer than ten, enabling inference speeds over ten times faster than diffusion or autoregressive models.
Training on the Imagen dataset with models ranging from 632M to 3B parameters, Muse achieves state-of-the-art FID scores of 6.06 on CC3M and 7.88 on zero-shot COCO, with a CLIP score of 0.32, outperforming prior models in both quality and efficiency. The model demonstrates a deep understanding of visual concepts such as object count, spatial relationships, pose, and style, translating complex language prompts into high-fidelity images. Importantly, Muse supports zero-shot image editing tasks like inpainting, outpainting, and style transfer, without additional fine-tuning, thanks to its masked training paradigm.
This work signifies a leap forward in scalable, high-quality text-to-image synthesis. By combining discrete token modeling, large-scale language understanding, and efficient parallel decoding, Muse addresses longstanding bottlenecks in generative AI. Its ability to generate detailed, diverse images rapidly opens new horizons for content creation, virtual reality, and AI-assisted design. Future research will focus on further accelerating inference, improving handling of complex prompts, and extending capabilities to video and 3D content, promising a broad impact across multiple industries.
Deep Analysis
Background
近年来，生成模型在文本到图像任务中取得了快速发展，代表性技术包括OpenAI的DALL-E、Google的Imagen以及Stable Diffusion。这些模型多基于扩散过程或自回归架构，虽然在图像质量方面不断突破，但在推理速度和多样性控制上仍存在瓶颈。Transformer架构的引入极大提升了多模态理解能力，结合大规模预训练模型（如GPT、T5）实现更细粒度的语义表达。此前的研究多集中在像素空间或连续潜在空间，缺乏高效的离散Token建模。Muse借鉴掩码建模和离散Token技术，结合预训练的语言模型，旨在突破生成速度与质量的矛盾，推动大规模高质量生成技术的发展。
Core Problem
现有的文本到图像生成模型在生成速度、细节还原和多样性方面仍有不足。扩散模型虽能生成高质量图像，但采样步骤繁琐，速度较慢；自回归模型虽快，但难以实现大规模并行。如何在保证图像质量的同时，大幅度提升推理效率，成为亟待解决的核心问题。此外，模型对复杂语言描述、多对象关系和细节理解能力仍需增强，以满足实际应用中的多样需求。
Innovation
1) 利用VQGAN编码图像为离散Token，减少连续空间的复杂性。2) 结合预训练大语言模型（T5-XXL）提取丰富的语义信息，增强模型对细粒度语言的理解。3) 引入变动掩码率策略，提升模型对不同遮挡比例的适应性。4) 采用并行解码机制，大幅缩短采样时间，实现快速推理。5) 通过多尺度生成架构，结合基础模型和超分辨率模型，支持高分辨率输出，兼顾细节和语义一致性。这些创新共同推动模型在效率和质量上的突破。
Methodology
- �� 使用预训练T5-XXL提取文本嵌入，作为Transformer模型的条件输入。• 利用两个VQGAN编码器，将图像编码为256×256和512×512的离散Token序列。• 基础掩码Transformer在Token空间中预测随机掩码的Token，训练中采用交叉熵损失。• 变动掩码率策略，随机遮挡不同比例Token，增强模型泛化能力。• 推理时，利用并行解码逐步预测Token，减少采样步骤（如24步预测256Token）。• 超分辨率Transformer将低分辨率Token转换为高分辨率Token，支持多尺度生成。• 支持无微调的图像编辑任务，通过掩码预测实现修补、扩展和无掩码编辑。
Experiments
模型在Imagen数据集上训练，632M和3B参数版本，训练时间约1周，采用TPU-v4。评估指标包括FID、CLIP得分和人类评价。CC3M上，632M模型FID达6.06，优于所有对比模型；在零样本COCO上，3B模型FID为7.88，CLIP得分0.32。对比Diffusion和自回归模型，Muse推理速度快10倍以上。还进行了多样性、风格和复杂场景的定性评估，验证模型对细粒度语言和多对象关系的理解能力。
Results
模型在多个指标上超越SOTA，尤其在FID和速度方面表现突出。CC3M上实现FID 6.06，显示出高质量生成能力。零样本COCO评估中，FID 7.88和CLIP 0.32表明模型对文本的理解和图像匹配能力强。推理速度比传统diffusion模型快10倍，支持实时交互。模型还能实现无微调的修补、扩展和风格迁移，展示了强大的多功能性。
Applications
模型可广泛应用于内容创作、虚拟现实、广告设计等领域，支持快速生成高质量图像。无需微调，用户只需提供文本描述即可获得对应图像，极大降低门槛。未来，结合多模态预训练和多任务学习，有望实现更复杂的场景理解和生成，推动AI在艺术、娱乐和工业设计中的创新。
Limitations & Outlook
模型在处理长句或复杂描述时，仍存在细节丢失和表达不准确的问题，尤其在高对象数量或长文本中表现不足。推理仍需大量计算资源，限制边缘设备部署。对极端风格或特殊场景的生成能力有限，未来需结合更丰富的数据和多模态信息进行优化。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂的任务是把文字变成图片。工厂里有一个特别的机器（模型），它先读懂你写的描述，就像工人用放大镜仔细看每个词。然后，它用一种特殊的“码”把图片拆成很多小块（Token），这些小块就像拼图的碎片。机器会随机把一些碎片遮住，然后猜出缺失的部分，就像拼图游戏一样。通过不断猜测和修正，最后拼出一幅完整的图片。这个工厂的特别之处在于，它可以非常快地完成拼图，比以前那些慢吞吞的机器快十倍，还能理解你描述的细节，比如对象的数量、位置和姿态。它还能在不需要重新训练的情况下，帮你修补、扩展图片，甚至改变风格，就像魔法一样。这个技术让我们未来可以用一句话，快速得到高质量的图片，应用在游戏、设计、虚拟现实等很多领域，变得更方便、更智能。
ELI14 Explained like you're 14
想象你在玩拼图游戏，但这个拼图可以自己动，知道你想要什么样的图片！你只要告诉它一句话，比如‘一只戴着太阳镜的猫’，它就能用很多小碎片拼出一幅漂亮的猫的画。这个拼图游戏特别厉害，因为它用了一种聪明的机器（模型），可以快速猜出缺失的碎片，而且还能理解你说的话里隐藏的细节，比如猫的姿势、背景或风格。更酷的是，它不用重新训练，只要给它一些例子，就能帮你修补图片、画出更大的画面，甚至改变画的风格。就像有个神奇的画家助手，听一句话就能画出你想要的图片，而且还很快！这让我们以后用一句话就能得到想要的图片，应用在游戏、动画、虚拟世界里，变得超级方便和有趣！
Abstract
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io