FonTS: Text Rendering with Typography and Style Controls
FonTS employs a two-stage DiT pipeline with parameter-efficient fine-tuning and style adapters to achieve precise word-level typography and style control, significantly improving text rendering quality.
Key Findings
Methodology
This work introduces a two-stage DiT-based pipeline combining parameter-efficient fine-tuning (TC-FT) and a text-agnostic style control adapter (SCA). The first stage fine-tunes only 5% of model parameters using enclosing typography control tokens (ETC-tokens) embedded via HTML-rendered data, enabling precise word-level font attribute manipulation. The second stage trains SCA modules to decouple style from content, enhancing style consistency without content leakage. The pipeline integrates multi-modal cues, balancing scene and artistic text rendering. The approach leverages a novel word-level controllable dataset, constructed through HTML rendering, to facilitate targeted learning of typographic features.
Key Results
- The model achieves 96.98% font consistency and 63.64% style consistency, with OCR accuracy reaching 82.85% on scene text benchmarks. In artistic text rendering, style coherence surpasses baselines like Flux and SD3, with significant improvements in word-level control. Quantitative metrics show superior control granularity, while qualitative assessments confirm precise attribute application. User studies further validate enhanced perceptual quality, demonstrating robustness across diverse scenarios.
- HTML-rendered data and ETC-tokens enable accurate learning of font attributes at the word level, reducing style drift. Fine-tuning only 5% of parameters ensures efficiency and preserves scene text capabilities. The decoupled attention modules prevent content leakage, improving style stability. Extensive ablation confirms the effectiveness of each component, with cross-scenario tests indicating strong generalization.
- Across multiple benchmarks, the proposed method outperforms existing models in font and style control, with lower error rates and higher consistency scores. The approach effectively handles complex typography and artistic styles, maintaining high OCR accuracy and visual coherence, even in challenging multi-style environments.
Significance
This research addresses a longstanding challenge in automated text rendering: achieving fine-grained, word-level control over font and style while maintaining overall scene rendering quality. By integrating parameter-efficient fine-tuning with multi-modal style adaptation, it offers a scalable solution for personalized and artistic text generation. The method's ability to produce consistent, high-quality text images has profound implications for digital content creation, branding, and creative industries, enabling rapid, customizable design workflows. Its technical innovations also open new avenues for multi-modal content synthesis, pushing the boundaries of AI-driven visual communication.
Technical Contribution
The core technical contribution lies in the development of a two-stage DiT pipeline that combines minimal parameter fine-tuning (only 5%) with a novel enclosing typography control token system, enabling precise word-level font attribute control. The HTML-rendered data pipeline facilitates high-quality, targeted dataset creation, critical for learning fine-grained typographic features. Additionally, the introduction of a decoupled joint attention mechanism within the style control adapter (SCA) effectively separates style from content, preventing leakage and improving style consistency. These innovations collectively enhance controllability, efficiency, and robustness of diffusion-based text rendering models, setting new standards in fine-grained visual control.
Novelty
This work is the first to realize word-level font and style control within a diffusion transformer framework, employing ETC-tokens for localized attribute manipulation. Unlike prior methods relying on full-parameter fine-tuning or coarse control, this approach fine-tunes only a small subset of parameters, significantly reducing computational costs. The integration of HTML-rendered datasets for targeted learning and the design of a text-agnostic style adapter further distinguish this work. These innovations collectively enable unprecedented control granularity and style stability, marking a significant leap forward in controllable image synthesis.
Limitations
- Despite improvements, the model struggles with extremely complex typography or highly stylized fonts, where attribute disentanglement becomes challenging. The reliance on high-quality datasets limits scalability in diverse real-world scenarios.
- Style transfer may still exhibit drift in highly artistic or abstract styles, especially when style images contain textual elements, risking content leakage despite the style adapter.
- Training and fine-tuning require substantial computational resources, and model efficiency could be further optimized for deployment in resource-constrained environments.
Future Work
Future directions include enhancing the model's ability to handle more complex and abstract typography, integrating self-supervised learning to reduce data dependency, and optimizing architecture for real-time applications. Expanding multi-modal fusion strategies could further improve style transfer fidelity. Additionally, exploring unsupervised or weakly supervised datasets may broaden applicability, while efforts to reduce computational costs will facilitate deployment in edge devices, enabling personalized, on-demand text generation.
AI Executive Summary
Visual text rendering has become a vital component in digital media, advertising, and branding. Traditional manual design methods, though flexible, are time-consuming and lack scalability. Recent advances in diffusion transformer (DiT) models have shown promise in automating this process, but they often fall short in achieving fine-grained control over individual words' typography and style. Existing models struggle with inconsistent fonts, style drift, and content leakage, especially at the word level.
To address these challenges, this paper introduces FonTS, a novel two-stage DiT-based pipeline that significantly enhances controllability. The first stage employs a parameter-efficient fine-tuning method called Typography Control Fine-tuning (TC-FT), which adjusts only 5% of the model's parameters. This stage leverages enclosing typography control tokens (ETC-tokens) embedded via an HTML-rendered data pipeline, enabling precise word-level font attribute manipulation. The second stage involves training a text-agnostic style control adapter (SCA), which decouples style from content, ensuring style consistency without content leakage.
The core innovation lies in combining these modules to achieve high-fidelity, controllable text rendering. Extensive experiments demonstrate that the proposed approach surpasses existing methods like Glyph-ByT5 and Flux in font consistency (96.98%) and style coherence (63.64%), while OCR accuracy on scene text reaches 82.85%. User studies confirm perceptual improvements, validating the effectiveness of the system across diverse scenarios.
This work advances the field by enabling detailed, word-level control over typography and style in automated text rendering, opening new possibilities for personalized content creation, artistic design, and digital branding. Its efficiency and controllability make it a promising foundation for future AI-driven visual content synthesis, with potential applications spanning advertising, digital art, and interactive media. Despite current limitations in handling highly stylized fonts and complex layouts, ongoing research aims to extend these capabilities, reduce computational costs, and broaden real-world deployment.
Deep Analysis
Background
随着数字内容的普及，视觉文本在广告、UI设计和数字艺术中的需求不断增长。早期方法多依赖手工排版和规则式生成，效率低且缺乏灵活性。近年来，扩散模型（如SD3、Flux）推动了自动化文本生成，但在字体一致性和细粒度控制方面仍存在瓶颈。现有DiT模型虽能生成艺术风格文本，但在字级控制和风格迁移方面表现不足，容易出现风格漂移和语义混淆。多模态融合和参数微调技术逐渐成为研究热点，为实现更精细的控制提供了可能。
Core Problem
当前文本渲染模型在实现字级字体属性调节和风格一致性方面存在明显不足。传统微调方法调节参数成本高，难以实现位置精确控制。风格迁移易引起内容泄露或风格漂移，影响生成质量。如何在保证场景文本能力的基础上，实现字级字体和风格的精准调控，成为亟待解决的难题。这不仅关系到内容的美观性，也影响到自动排版和个性化设计的实现。
Innovation
本研究提出结合参数微调（TC-FT）和封闭式排版控制标记（ETC-tokens）的两阶段DiT管线，实现在字级别的字体属性调节。引入HTML渲染技术，构建字级数据集（TC-Dataset），提升模型对字体细节的学习能力。设计风格适配器（SCA），解耦内容与风格，有效防止内容泄露，增强风格一致性。创新点还包括只调节模型少量参数，显著提升效率，兼顾场景文本和艺术文本的控制需求。这些创新共同推动了细粒度、稳定的文本生成技术。
Methodology
- �� 构建两阶段DiT管线：第一阶段进行参数高效微调（TC-FT），只调节模型5%的参数，嵌入ETC-tokens实现字级属性控制；第二阶段训练风格适配器（SCA），解耦内容和风格，增强风格一致性。
- �� 设计封闭式排版控制标记（ETC-tokens），在文本中嵌入起止标记，精准调节字体粗细、斜体、下划线等属性。
- �� 利用HTML渲染技术，合成多样化高质量字级数据集（TC-Dataset），提升模型对字级属性的学习能力。
- �� 结合多模态信息，优化模型在场景文本和艺术文本中的控制能力。
- �� 微调后，利用SCA模块解耦内容与风格，避免内容泄露，确保风格稳定。
Experiments
采用自制的字级控制数据集（TC-Dataset）和风格数据集（SC-general、SC-artext），在基本文本（BTR）和艺术文本（ATR）基准上评估。指标包括字体一致性（FontCLIP-I）、风格一致性（Style-Con）和OCR识别率（OCR-Acc）。对比Glyph-ByT5、Flux等模型，进行AB测试和用户评估。训练在8×A100上进行，步数分别为40k（TC-FT）和100k（SCA），确保多场景泛化。
Results
模型在字级字体控制中，字体一致性达96.98%，风格一致性63.64%，OCR识别率82.85%。在艺术文本中，风格稳定，内容准确。用户评估显示优越控制效果。AB实验验证，参数微调和ETC-tokens显著提升控制精度，风格适配器缓解内容泄露。多场景测试表明模型鲁棒性强，能应对复杂字体和风格变化。
Applications
该技术适用于广告、个性化字体设计、数字排版和艺术创作。用户只需输入文本和偏好，即可自动生成高质量文本图像。未来结合实时交互和边缘计算，有望实现动态个性化排版，极大提升内容创作效率。
Limitations & Outlook
模型在极端字体和复杂排版场景中仍存在控制不足，部分字体属性调节受限。风格迁移在极端风格下可能漂移，且训练依赖大量高质量数据，成本较高。未来需优化模型结构，降低计算成本，增强泛化能力。
Plain Language Accessible to non-experts
想象你在做一本书的排版。以前，编辑需要花很多时间挑选字体、调整样式，费时又繁琐。现在，有了智能排版助手，它可以根据你的指令，自动帮你调整每个单词的字体、粗细、斜体或下划线。这个助手通过学习大量排版样例，理解不同字体和风格的搭配，然后精准地在每个单词上应用效果。它还能确保整本书的风格一致，不会出现字体乱七八糟的情况。这样一来，设计师可以节省很多时间，作品也更漂亮。这就像一个聪明的排版机器人，帮你把文字变得更有趣、更专业。
ELI14 Explained like you're 14
想象你在用电脑做一个海报，你可以告诉它：“让‘快乐’这个词变得又粗又大，颜色是红色，还带点斜体。”以前，要做到这些需要花很多时间调字体、调整样式，但现在有了这个新技术，它可以一秒钟帮你搞定！这个系统学会了很多不同的字体和风格，然后根据你的指令，把每个单词变成你想要的样子。它还能保证整个海报的风格一致，不会出现乱七八糟的字体。就像你有一个超级厉害的设计助手，帮你快速做出漂亮的作品。未来，这个技术还能帮广告公司、设计师们节省大量时间，让每个人都能轻松做出酷炫的文字效果！
Abstract
Visual text rendering are widespread in various real-world applications, requiring careful font selection and typographic choices. Recent progress in diffusion transformer (DiT)-based text-to-image (T2I) models show promise in automating these processes. However, these methods still encounter challenges like inconsistent fonts, style variation, and limited fine-grained control, particularly at the word-level. This paper proposes a two-stage DiT-based pipeline to address these problems by enhancing controllability over typography and style in text rendering. We introduce typography control fine-tuning (TC-FT), an parameter-efficient fine-tuning method (on $5\%$ key parameters) with enclosing typography control tokens (ETC-tokens), which enables precise word-level application of typographic features. To further address style inconsistency in text rendering, we propose a text-agnostic style control adapter (SCA) that prevents content leakage while enhancing style consistency. To implement TC-FT and SCA effectively, we incorporated HTML-render into the data synthesis pipeline and proposed the first word-level controllable dataset. Through comprehensive experiments, we demonstrate the effectiveness of our approach in achieving superior word-level typographic control, font consistency, and style consistency in text rendering tasks. The datasets and models will be available for academic use.