One Rewrite to Fix Them All? Type-Aware Repair Allocation for Text-to-Image Prompt Optimization
TARA employs type-aware repair allocation, boosting semantic accuracy by +5.6 points on DSG, +2.6 on TIFA, with no retraining needed.
Key Findings
Methodology
TARA introduces a four-stage process: diagnosing atomic propositions using DSG-based decomposition, routing failures to type-conditioned local repairs, compiling heterogeneous repairs into a single prompt via a text portfolio, and adopting the repaired image only if semantic scores improve. It leverages fixed repair operators, a semantic repair gate, and a single generation budget, avoiding training or fine-tuning. The approach separates failure diagnosis, repair routing, prompt compilation, and acceptance, ensuring precise handling of diverse failure types. Key algorithms include atomic proposition detection, type mapping, repair constraint formulation, and text fusion, all within an end-to-end, training-free pipeline.
Key Results
- Across DSG and TIFA datasets, TARA outperforms baselines in semantic accuracy, with improvements of 5.6 and 2.6 points respectively, across four models (SD v1.5, SD v2.1, Flux-dev, Janus-Pro). It maintains image quality, with runtime reduced to 16 seconds per prompt versus 20 seconds for baselines. Extensive ablations confirm the importance of type-conditioned routing, repair compilation, and the semantic gate. The method shows robustness across different semantic categories, especially in complex scenes involving counts, relations, and text.
- Results demonstrate that explicit type-aware repair significantly surpasses uniform expansion strategies, especially in scenarios with heterogeneous failures. The ablation studies reveal that each component—failure diagnosis, repair routing, compilation, and gating—contributes to the overall performance. The evaluation metrics include semantic accuracy, CLIP score, aesthetic score, and human preference, all favoring TARA. The approach generalizes well across multiple datasets and models, confirming its broad applicability.
- Furthermore, TARA achieves a balanced trade-off between semantic fidelity and image aesthetics, with CLIP scores comparable to raw prompts. Its efficiency and reliability make it suitable for real-world deployment, enabling content creators and AI systems to generate more accurate and reliable images without retraining or extensive tuning.
Significance
This work advances prompt optimization by introducing a type-aware, atomic repair framework that addresses the limitations of uniform expansion strategies. It enables precise, heterogeneous failure handling, significantly improving semantic fidelity in text-to-image generation. The approach is training-free, model-agnostic, and computationally efficient, making it highly practical for diverse applications. Its ability to reliably correct complex semantic errors paves the way for more trustworthy AI-generated content, impacting fields like digital art, virtual reality, and automated content moderation. By systematically routing failures to specialized repairs, TARA sets a new standard for semantic robustness in multimodal generation, fostering future research into fine-grained, adaptive prompt engineering.
Technical Contribution
The paper introduces a novel framework that decomposes prompt failures into atomic propositions, classifies them into predefined types, and assigns dedicated, fixed repair operators. It employs a text-only portfolio to compile heterogeneous local repairs into a single prompt, ensuring semantic coherence. The semantic repair gate acts as a safeguard against regressions, only accepting repairs that improve semantic scores. This approach departs from prior methods relying on uniform expansion or learned routers, offering a transparent, interpretable, and training-free solution. The integration of failure diagnosis, routing, compilation, and decision-making into a cohesive pipeline represents a significant technical innovation, enabling precise, multi-failure correction within a single generation cycle.
Novelty
This is the first work to explicitly decompose visual feedback failures into multiple semantic categories and route each to a dedicated, fixed repair operator. Unlike previous approaches that treat all failures uniformly, TARA’s type-conditioned routing ensures targeted corrections, significantly improving semantic accuracy. Its use of a text-only portfolio for repair compilation and a semantic gate for reliable adoption further distinguishes it from existing methods, which often rely on heuristic or end-to-end learned models. This granular, interpretable, and training-free approach marks a new direction in prompt optimization and multimodal content refinement.
Limitations
- TARA’s fixed type mapping may struggle with novel or ambiguous failure types not covered by predefined categories, limiting adaptability in highly diverse scenarios. Its reliance on visual feedback quality means that errors in failure detection can propagate, affecting repair accuracy. The process, while efficient, still involves multiple steps that could be optimized further, especially for real-time applications. Additionally, in cases of severe ambiguity or multi-failure interactions, the single-pass repair might not fully resolve all issues, necessitating iterative refinement or user intervention.
Future Work
Future directions include integrating learning-based repair routing to adaptively handle unseen failure types, enhancing failure detection robustness, and optimizing the prompt compilation process for faster inference. Extending the framework to multi-turn interactions, where user feedback guides repair priorities, could improve usability. Exploring multi-failure joint correction strategies and incorporating multimodal feedback (e.g., audio, user annotations) are promising avenues. Additionally, applying TARA’s principles to other multimodal tasks like video synthesis or 3D scene generation could broaden its impact.
AI Executive Summary
Despite rapid advancements in text-to-image generation, achieving faithful semantic alignment remains a challenge. Existing prompt optimization techniques often rely on uniform prompt expansion, which inadequately addresses the diverse nature of semantic failures such as object omission, miscount, relation errors, and illegible text. These methods tend to treat all failures uniformly, leading to suboptimal corrections and persistent errors in complex scenes. Recognizing this limitation, the authors propose TARA, a novel, training-free framework that introduces type-aware repair allocation. TARA decomposes the prompt failures into atomic propositions, diagnoses their types, and routes each to a dedicated, fixed repair operator. This targeted approach ensures that each failure receives a precise correction, whether it involves adding objects, adjusting spatial relations, or clarifying text. The repaired local constraints are then compiled into a single, coherent prompt via a text-only portfolio, which balances different repair biases. To prevent regressions, a semantic repair gate evaluates the improvedness of the generated image before adoption, ensuring semantic fidelity. Extensive experiments across multiple datasets, models, and semantic categories demonstrate that TARA consistently outperforms existing methods like VisualPrompter, with an average semantic accuracy gain of 5.6 points on DSG and 2.6 on TIFA, while maintaining image quality and reducing runtime to 16 seconds per prompt. This approach marks a significant step toward more reliable, interpretable, and adaptable prompt optimization in multimodal AI systems. Its modular, model-agnostic design opens avenues for broader applications, including personalized content creation, complex scene understanding, and real-time interactive generation, fostering a future where AI-generated visuals are both high-fidelity and semantically trustworthy.
Deep Analysis
Background
文本到图像（T2I）技术经历了从早期基于规则到深度学习模型的快速演进，代表性工作如DALL·E、Stable Diffusion等极大推动了自动内容生成。尽管如此，生成模型在语义一致性方面仍存在挑战，尤其是在复杂场景、多对象关系和文本信息的准确表达方面。现有优化策略多依赖提示微调或后处理技术，如VisualPrompter和Promptist，但这些方法多采用全局统一扩展，难以应对多样化的失败类型。近年来，视觉反馈引导的提示优化逐渐兴起，试图通过检测生成内容中的错误，动态调整提示，但缺乏对失败类别的细粒度分类和针对性修复机制。本文基于此背景，提出一种类型感知的修复策略，旨在解决多样化语义故障的精准修复难题，为提升生成内容的语义一致性提供新思路。
Core Problem
当前文本到图像生成中，模型经常出现错漏、关系错乱、文本不清晰等问题。传统方法多采用全局扩展，忽视不同失败类型的差异，导致修复效果有限，甚至引入新的语义偏差。这种一刀切策略在复杂场景下表现尤为不足，限制了内容的准确性和应用的可靠性。如何实现多类型失败的精准识别与定制修复，成为提升生成质量的关键。尤其在多模态内容融合、复杂关系表达等方面，现有方法难以满足需求。解决这一问题，不仅需要细粒度的故障诊断，还需设计类型感知的修复策略，确保修复的语义一致性和系统的鲁棒性。
Innovation
本研究的核心创新在于提出基于视觉反馈的原子命题失败细分与类型感知修复机制。具体包括：1）将失败的原子命题按语义类别映射到不同修复操作，实现异质失败的精准路由；2）设计无训练的局部修复策略，结合文本融合机制，将多类型修复统一编译成单一提示；3）引入语义修复门控，确保修复后内容的语义提升而非回退。这一策略区别于传统单一扩展方法，显著增强了对复杂场景的适应能力。其创新点在于将视觉反馈中的失败细分为多类别，并据此定制修复方案，从而实现更高的语义准确率和系统鲁棒性。
Methodology
- �� 诊断阶段：利用DSG方法，将提示分解为原子命题，基于视觉语言模型（VLM）对生成图像进行语义验证，识别出错误或缺失的内容。• 失败路由：将诊断结果中的失败原子命题映射到预定义的类型（如缺失对象、错误计数、关系错乱等），作为路由变量。• 修复分配：为每个失败类型选择对应的局部修复操作（如添加对象、调整位置、修正文本），确保修复具有针对性。• 编译融合：采用文本候选集（如全扩展、类型标注扩展、最小补充）将局部修复合成为单一提示，避免修复间的冲突。• 采纳机制：生成修复图像后，基于语义门控判断是否采纳，确保修复带来语义提升，否则回退到原始内容。• 关键算法包括基于DSG的原子命题检测、类型映射、文本融合策略和门控决策，整体流程无需训练，端到端实现。
Experiments
在DSG-1k和TIFA数据集上，采用三种不同随机种子，设计了多模型（如SD v1.5、SD v2.1、Flux-dev、Janus-Pro）对比多种提示优化方法。指标包括语义准确率、CLIP得分和美学评分，评估修复效果和图像质量。通过AB测试和消融实验，验证了类型感知修复的有效性。参数设置方面，门控阈值设为0.72，修复候选集包括全扩展、类型标注扩展和最小补充，确保在有限生成预算下实现最大提升。对比基线包括Promptist、BeautifulPrompt、NeuroPrompts、TIPO和VisualPrompter，所有方法在相同条件下进行评估，确保公平性。
Results
TARA在所有八个基准-生成器组合中均实现最高语义准确率，平均提升5.6点（DSG）和2.6点（TIFA），显著优于VisualPrompter。在不同生成模型和数据源中，TARA表现稳定，尤其在复杂语义场景（如数量、关系、文本）中优势明显。AB测试显示，TARA的修复速度快，平均16秒，优于20秒基线。消融分析确认类型划分、修复编译和门控机制的贡献，确保修复的语义一致性和鲁棒性。多指标评估也表明，TARA在保持图像质量的同时，显著提升语义准确性。
Applications
该方法适用于需要高语义一致性的内容生成场景，如广告设计、虚拟现实、游戏开发和自动内容审核。无需模型微调，便于在现有生成系统中快速部署，提升内容的准确性和用户体验。未来还可结合用户反馈，动态调整修复策略，实现个性化定制，满足多样化需求。
Limitations & Outlook
尽管TARA在多场景下表现优异，但在极端模糊或多义性强的提示中仍可能出现修复不足。其依赖固定类型映射，面对新颖或未覆盖的失败类型时效果有限。修复过程受限于生成次数和文本融合的效率，未来需优化算法和模型结构，以应对更大规模和复杂场景的需求。
Plain Language Accessible to non-experts
想象你在厨房做菜，遇到食材不新鲜或调料用错了。传统方法可能只会多放点调料或换个菜谱，但不考虑具体问题。TARA就像一个聪明的厨师，能识别每个问题，比如盐放少了、火候不对，然后用不同的技巧修正。它会先检查菜肴哪里出错，然后根据问题类型，选择不同的修正方法，比如多放盐、调节火候，最后合成一道美味佳肴。这样，每个问题都能被精准解决，菜肴也变得更好吃。这种分门别类的修正方式，比一刀切的办法更聪明、更有效。它让厨房里的菜变得更完美，也能用在生成图片上，让图片更符合描述。
Abstract
Text-to-image (T2I) generators often fail to follow their prompts faithfully, producing wrong counts, swapped attributes, ambiguous relations, and illegible text. Prompt optimization repairs such failures by rewriting the user prompt, requiring no generator retraining, and has yielded promising results. However, existing optimizers absorb heterogeneous failures into one uniform prompt expansion, even though each calls for different repair language. We formulate semantic prompt optimization as atomic repair allocation: each failed proposition is routed to a type-conditioned repair operator before the resulting local constraints are compiled into one executable prompt. We instantiate this formulation in the training-free Type-Aware Repair Allocation (TARA) framework, which separates diagnosis, allocation, compilation, and a semantic repair gate, an accept-or-revert controller over exactly one prescribed repair that prevents semantic regressions. Extensive experiments on DSG and TIFA across four frozen generators demonstrate that TARA achieves the best semantic accuracy in all eight benchmark-generator cells, improving over VisualPrompter by 5.6 and 2.6 points on DSG and TIFA, respectively, while maintaining image quality and running fastest in our matched local setting at 16.0 seconds versus 20.0 seconds per prompt.