Optimizing Prompts for Text-to-Image Generation
Proposed PROMPTIST, a reinforcement learning framework, improves text-to-image prompts, boosting reward by over 30% on Stable Diffusion.
Key Findings
Methodology
The approach combines supervised fine-tuning and reinforcement learning, leveraging a pretrained language model (e.g., GPT) to adapt manually engineered prompts. The fine-tuned model initializes a prompt policy network, which is further optimized via PPO. The reward function integrates CLIP-based relevance and aesthetic scores, guiding the policy to generate prompts that produce more relevant and visually pleasing images. Key steps include:• Collecting prompt pairs for supervised training;• Using diverse beam search for prompt diversity;• Sampling images conditioned on prompts, computing relevance and aesthetic scores;• Optimizing prompts through RL to maximize combined rewards. Experiments on Stable Diffusion validate the method, outperforming manual prompts in automatic metrics and human preferences.
Key Results
- On Stable Diffusion v1.4 and v1.5, optimized prompts yield over 30% higher reward scores compared to manual prompts, with reinforcement learning further boosting performance, especially on out-of-domain prompts, with improvements up to 127%. Relevance scores increased from 0.25 to 0.26, and aesthetic scores improved significantly, reaching 6.26 out of 7. Human evaluations show a preference for images generated with optimized prompts, with over 70% favorability. These results demonstrate the method’s robustness and generalization, especially in unseen domains.
- Reward improvements are consistent across datasets like DiffusionDB and COCO, with reinforcement learning providing larger gains on out-of-domain data. Source prompt augmentation enhances model generalization, enabling better adaptation to diverse inputs. The reward function effectively balances relevance and aesthetics, guiding the model toward higher-quality prompts. Ablation studies confirm the importance of RL and data augmentation strategies in achieving optimal results.
- The core innovation lies in combining supervised fine-tuning with RL, utilizing multi-objective rewards based on CLIP relevance and aesthetic scores. This enables automatic prompt optimization within black-box models, surpassing traditional manual engineering and paraphrasing methods, and providing a scalable solution for improving text-to-image generation quality.
Significance
This work addresses the challenge of prompt sensitivity in text-to-image models, offering a systematic, automated approach to generate high-quality prompts aligned with user intent. It reduces reliance on manual engineering, enhances model robustness across domains, and accelerates the deployment of personalized, high-fidelity visual content. The framework bridges the gap between user-friendly inputs and model-preferred prompts, facilitating broader adoption in creative industries, virtual environments, and AI-assisted art. It also opens avenues for integrating human feedback more effectively into generative systems, advancing the state-of-the-art in multimodal AI.
Technical Contribution
The paper introduces a novel framework combining supervised fine-tuning and reinforcement learning, utilizing a multi-objective reward function that integrates CLIP relevance and aesthetic scores. It innovates by applying PPO for prompt policy optimization in black-box models, employing diverse beam search for exploration, and source prompt augmentation for better generalization. These contributions enable automatic, scalable prompt refinement, significantly improving image quality and relevance, and setting a new standard for prompt engineering in generative AI.
Novelty
This research is the first to systematically integrate supervised fine-tuning with reinforcement learning for prompt optimization in text-to-image models. Unlike prior methods relying solely on manual design or paraphrasing, it employs a multi-objective reward to guide prompt generation, enabling automatic adaptation to diverse inputs and domains. Its end-to-end framework and use of PPO in a black-box setting represent a significant advancement over existing prompt search and continuous prompt methods, establishing a new paradigm for prompt engineering.
Limitations
- 依赖预训练模型（如GPT和CLIP），在特定专业或风格领域可能表现不足，泛化能力有限。
- 奖励函数主要基于自动评分指标，可能无法完全捕捉用户的主观偏好，存在偏差。
- 训练过程计算成本较高，难以实现实时或大规模应用，限制了实际部署的可能性。
Future Work
未来将探索引入用户反馈进行动态调整，提升个性化和多样性。结合多模态奖励机制，增强模型对复杂场景的适应能力。同时，优化训练效率，降低计算成本，也是重要方向。
AI Executive Summary
随着深度生成模型的不断发展，提示设计成为影响文本到图像生成效果的关键因素。传统的手工提示工程耗时且难以迁移，限制了模型的普及和应用。为解决这一瓶颈，本文提出PROMPTIST，一种结合强化学习的自动提示优化框架，旨在将用户输入自动转化为模型偏好的提示。该方法首先利用有监督微调，基于手工提示数据训练语言模型，然后通过PPO强化学习不断探索更优提示。奖励函数结合CLIP相关性和美学评分，确保生成的图像既符合用户意图，又具审美价值。实验在Stable Diffusion上验证，结果显示优化提示在自动指标和人类偏好中均优于手工提示，尤其在域外数据中表现出强大泛化能力。这一技术突破了传统提示工程的瓶颈，为多模态生成系统的自动化和智能化提供了新思路。未来，结合用户反馈和多模态奖励机制，将进一步提升模型的个性化和适应性，推动生成艺术、虚拟现实等行业的创新发展。
Deep Analysis
Background
近年来，生成模型，尤其是文本到图像（text-to-image）技术，取得了突破性进展。从GANs（如StackGAN）到自回归模型（如DALL·E）和扩散模型（如Stable Diffusion），生成质量不断提升。提示工程作为关键环节，传统依赖人工设计，耗时且难以迁移，限制了模型的普及。自动提示搜索、paraphrasing和连续提示优化等方法逐渐出现，但仍面临泛化不足和效率瓶颈。本研究旨在通过强化学习实现提示的自动化优化，突破现有限制。
Core Problem
文本到图像模型对提示的敏感性较高，手工工程难以满足多样化需求，且缺乏通用性。手工提示难以适应不同模型版本和新领域，导致生成效果不稳定。如何自动生成高质量、符合用户意图的提示，成为亟需解决的问题。此外，模型在域外数据上的表现不足，限制了其实际应用范围。
Innovation
本研究的主要创新包括：• 结合有监督微调和强化学习，自动优化提示；• 设计多目标奖励函数，结合CLIP相关性和美学评分，平衡相关性与审美；• 引入源提示增强策略，提升模型泛化能力。这些创新突破了传统手工工程的局限，实现了提示的端到端自动化，显著提升生成效果。
Methodology
- �� 收集手工提示对，构建平行数据集，进行微调；• 利用多样性束搜索生成多样提示，保证探索空间；• 采样图像，计算CLIP相关性和美学评分，作为奖励指标；• 设计奖励函数，结合相关性、美学和KL正则项，平衡目标；• 采用PPO算法，优化策略参数，持续探索更优提示。整个流程实现了从用户输入到模型偏好的自动转化，确保提示既符合用户意图，又能引导模型生成高质量图像。
Experiments
在Stable Diffusion v1.4和v1.5上进行验证，使用DiffusionDB、COCO和ImageNet-21k数据集。微调阶段采集90k手工提示，强化学习阶段探索域外数据。评估指标包括自动奖励（相关性和美学得分）和人类偏好。超参数设置包括：批次256，学习率5e-5，束宽8，最大生成长度15-75。对比手工提示、微调模型和RL模型，验证其在多场景下的优越性。
Results
优化提示在奖励和偏好指标上均优于手工提示，平均提升超过30%。在域外数据中，强化学习模型表现尤为突出，奖励提升达127%。相关性由0.25提升至0.26，美学评分显著改善。人类评测显示，用户偏好由优化提示生成的图像比例超过70%。这些结果验证了方法的有效性和强泛化能力，特别是在未见过的域外场景中表现优异。
Applications
该方法可广泛应用于虚拟艺术、广告设计、虚拟现实等领域，降低专业门槛，提升生成效率。用户只需提供简要描述，系统即可自动生成高质量提示，简化操作流程。未来结合个性化偏好，打造智能化、多样化的内容生成平台，推动多模态内容产业发展。
Limitations & Outlook
模型依赖预训练模型（如GPT和CLIP），在特定风格或专业领域表现可能不足。奖励函数主要基于自动评分指标，难以完全反映用户主观偏好。训练成本较高，限制了实时应用和大规模部署。未来需优化算法效率，丰富奖励机制，增强个性化和多样性。
Plain Language Accessible to non-experts
想象你在厨房做菜，你告诉厨师你想吃什么，比如“我想吃一只蓝色的狼”。传统上，厨师会根据你的描述用很多调料和步骤来做菜，但每次都需要你亲自指导。现在，有个聪明的机器人厨师，它可以听懂你的大意，然后自己试着调配调料，不断尝试，直到做出你喜欢的味道。它会学习你喜欢的口味（比如喜欢辣一点或甜一点），还会根据味道好不好（用评分衡量）不断改进。这样，你只要告诉它“我想吃一只蓝色的狼”，它就能自动调出最符合你心意的菜肴。这个机器人厨师就像一个聪明的助手，帮你省时省力，还能变得越来越会做你喜欢的菜。这就像让机器自己学会调配“菜谱”，让你的生活变得更方便、更有趣。
ELI14 Explained like you're 14
想象你喜欢画画，但不太会用画笔。你告诉一个智能机器人“我想画一只蓝色的狼”，它会试着自己画出不同版本。有时候画得不太像你想的，但它会学习你的喜好，慢慢改进。它会用一种叫“强化学习”的方法，就像你给它评分：画得漂亮就给高分，不满意就给低分。这样，它不断尝试，最后画出的狼既符合你的描述，又很漂亮。它还会记住你喜欢的风格，比如“像漫画一样”或者“很写实”，帮你画出更满意的作品。它就像一个聪明的画家助手，能帮你快速得到理想的画作，不用自己费心琢磨。这个方法让机器变得更聪明，也让我们的创作变得更轻松有趣。
Abstract
Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental results on Stable Diffusion show that our method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. Moreover, reinforcement learning further boosts performance, especially on out-of-domain prompts. The pretrained checkpoints are available at https://aka.ms/promptist. The demo can be found at https://aka.ms/promptist-demo.