TextDiffuser: Diffusion Models as Text Painters
TextDiffuser combines Transformer layout prediction with latent diffusion models, enabling high-quality, controllable text image synthesis.
Key Findings
Methodology
This paper introduces TextDiffuser, a two-stage framework: first, a Transformer predicts keyword layouts and character-level masks from text prompts; second, a latent diffusion model generates images conditioned on these masks and prompts. The approach employs CLIP encoding, Transformer-based layout prediction, and character-aware loss to enhance text rendering. The large-scale MARIO-10M dataset with OCR annotations supports training, enabling precise control over text placement and appearance. During inference, the model supports prompt-based, template-based, and inpainting tasks, offering flexible control over generated images.
Key Results
- On the MARIO-Eval benchmark, TextDiffuser achieves a CLIPScore of 0.3436, OCR accuracy of 78.06%, and FID of 38.758, outperforming StableDiffusion and DeepFloyd in text consistency and visual quality.
- Ablation studies show that using two Transformer layers, width embeddings, and character-aware loss significantly improves layout IoU (up to 0.298) and OCR accuracy (up to 49.4%).
- Qualitative and user studies demonstrate superior control and readability in complex scenes, multi-line texts, and background coherence, surpassing existing methods like ControlNet and GlyphDraw.
Significance
This work advances the state-of-the-art in text rendering within diffusion-based image generation, addressing longstanding issues of text clarity and layout control. The large-scale annotated dataset and comprehensive benchmark set a new standard for future research, enabling more accurate, controllable, and diverse text image synthesis. Its industrial implications span advertising, publishing, and digital content creation, reducing manual effort and improving visual quality in practical applications.
Technical Contribution
The paper's key innovations include integrating Transformer-based layout prediction with character-aware loss into a latent diffusion framework, supported by a large OCR-annotated dataset. This combination allows explicit character-level control, high fidelity text rendering, and flexible inpainting. The approach introduces a novel multi-task training scheme and a character-level segmentation guidance, enabling precise placement and style of text in complex backgrounds, setting a new benchmark in controllable diffusion-based text synthesis.
Novelty
This research is the first to combine Transformer layout prediction with latent diffusion models for detailed, multi-line text rendering in images. Unlike prior works relying solely on text encoders or simple conditioning, it achieves explicit character-level control and high accuracy, supported by a large OCR-annotated dataset. The integration of layout prediction, character-aware loss, and diffusion in a unified framework represents a significant leap forward in controllable text image synthesis.
Limitations
- The model struggles with extremely complex backgrounds or very long texts, where OCR errors can affect rendering quality. Its performance on multilingual or highly stylized fonts needs further improvement.
- Training requires substantial computational resources, limiting real-time applications and scalability.
- Generalization to unseen languages or fonts remains limited; future work should incorporate multi-modal data and more diverse training sets to enhance robustness.
Future Work
Future directions include enhancing multilingual and multi-font capabilities, reducing computational costs, and improving robustness in challenging scenarios. Integrating multi-modal cues like audio or sketches could further expand controllability. Additionally, scaling up training data with more diverse styles and backgrounds will help the model generalize better for real-world applications such as automated advertising, digital publishing, and interactive design tools.
AI Executive Summary
The rapid progress of diffusion models has revolutionized image synthesis, yet generating accurate, coherent text within images remains a significant challenge. Traditional approaches often produce blurry or misspelled characters, especially against complex backgrounds, limiting their practical utility in advertising, publishing, and design. To address this, the authors introduce TextDiffuser, a novel framework that combines Transformer-based layout prediction with latent diffusion models, enabling precise control over text placement and style.
The core innovation lies in a two-stage process: first, a Transformer predicts the spatial layout of keywords and generates character-level segmentation masks from text prompts, leveraging a large-scale OCR-annotated dataset. Second, a latent diffusion model synthesizes images conditioned on these masks and prompts, with a character-aware loss ensuring high-quality text rendering. This design allows for flexible tasks, including text-to-image generation, template-guided synthesis, and inpainting, significantly enhancing controllability.
Experimental results demonstrate that TextDiffuser outperforms existing models like Stable Diffusion and DeepFloyd on the MARIO-Eval benchmark, achieving superior CLIPScore, OCR accuracy, and visual coherence. Ablation studies confirm the importance of layout prediction layers, width embeddings, and character-aware loss. User studies further validate its ability to generate clear, natural, and contextually appropriate text, even in complex scenes.
This work marks a substantial step forward in AI-driven text rendering, providing a robust, scalable solution for high-quality, controllable image synthesis. Its implications extend across industries, promising to streamline creative workflows and democratize high-fidelity visual content creation. Future research will focus on multilingual support, efficiency improvements, and broader application scenarios, aiming to realize fully autonomous, versatile text-image generation systems.
Deep Dive
Abstract
Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. The code, model, and dataset will be available at \url{https://aka.ms/textdiffuser}.