Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Proactive T2I agent uses belief graphs and multi-turn questioning to improve alignment, achieving 2x VQAScore over standard methods.
Key Findings
Methodology
This paper introduces a belief graph-based proactive T2I agent that employs large language models (LLMs) to facilitate multi-turn interactions. The agent constructs a symbolic belief state capturing entities, attributes, and relations with associated probabilities and importance scores. Using a combination of heuristic scoring and belief-driven question generation, it actively identifies the most uncertain and impactful elements to query the user. The process involves iterative belief updates based on user feedback, guiding the prompt refinement. An automated evaluation pipeline simulates user intent via ground-truth images, measuring performance with metrics like VQAScore across datasets such as ImageInWords, COCO, and DesignBench. Results demonstrate at least 2x improvement over single-turn baseline in image quality and alignment.
Key Results
- Across datasets, the proposed agent achieves over 0.8 VQAScore within 5 interaction turns, doubling the performance of traditional single-turn methods. It effectively reduces uncertainty, leading to more accurate image generation aligned with user intent.
- Human studies show 90% of participants find the belief graphs and questioning helpful, with generated images surpassing baseline quality in over 80% of cases. The system significantly shortens the iterative refinement process.
- Automated simulations confirm that active questioning reduces interaction rounds needed for high-fidelity outputs, demonstrating robustness across diverse prompts and styles.
Significance
This work advances AI's capacity for active understanding and collaboration, moving beyond passive prompt-based generation. By explicitly modeling uncertainty with belief graphs and enabling proactive clarification, it enhances interpretability, user control, and output quality. The approach addresses long-standing challenges in aligning AI outputs with complex, underspecified human intents, especially in artistic and design contexts. It paves the way for more accessible, responsible, and user-friendly generative AI systems, fostering broader adoption in creative industries and human-AI collaboration.
Technical Contribution
The paper introduces a modular framework combining symbolic belief graphs with LLM-driven question generation, enabling dynamic uncertainty reduction. It develops a scalable automatic evaluation pipeline, integrating simulated user intent and image similarity metrics. The design supports flexible integration with various T2I models, and the proposed question selection strategies—heuristic and belief-driven—significantly improve interaction efficiency. This work bridges symbolic reasoning and deep learning, offering a novel paradigm for multi-turn generative AI systems.
Novelty
This is the first work to incorporate belief graphs for explicit uncertainty modeling in multi-turn text-to-image generation, enabling models to actively identify and query ambiguous or impactful elements. Unlike prior passive models, it employs a systematic question-asking strategy guided by importance and uncertainty scores, leading to more efficient and accurate alignment. The combination of symbolic belief representation with large language models for iterative refinement represents a significant innovation in the field.
Limitations
- The belief graph construction relies heavily on LLM accuracy; errors in parsing or inference can mislead the questioning process. In complex scenes, the system may still miss critical uncertainties.
- The current approach assumes simulated user responses; real-world interactions may introduce variability and noise, affecting robustness.
- Computational costs increase with multi-turn interactions and belief updates, which may limit scalability in real-time applications.
Future Work
Future directions include integrating multimodal belief representations, enhancing user modeling for personalized interactions, and applying reinforcement learning to optimize question strategies. Expanding to real user studies will improve robustness, while reducing computational overhead will facilitate deployment in interactive platforms. Additionally, extending the framework to other generative tasks like video synthesis or 3D modeling is envisioned.
AI Executive Summary
This research introduces a novel proactive agent for multi-turn text-to-image (T2I) generation, addressing the persistent challenge of prompt under-specification. Traditional models operate passively, often requiring users to iteratively refine prompts, which is inefficient and frustrating. The proposed solution leverages a belief graph—a symbolic, probabilistic representation of entities, attributes, and relations—combined with large language models (LLMs) to enable the agent to actively identify uncertainties and ask targeted clarification questions.
The core innovation lies in the agent’s ability to dynamically assess which elements are most uncertain and impactful, guided by importance scores and entropy measures. Using a modular architecture, the agent formulates questions—either multiple-choice or open-ended—aimed at reducing ambiguity. User responses update the belief graph, refining the prompt for the subsequent image generation. This iterative process significantly improves the alignment between generated images and user intent.
Extensive experiments on datasets like ImageInWords, COCO, and a curated DesignBench demonstrate that the agent achieves over twice the VQAScore of standard single-turn systems within five interaction rounds. Automated evaluation pipelines, simulating user intent, confirm the robustness and efficiency of the approach. Human studies further validate the system’s practical benefits, with 90% of participants rating the belief graphs and questioning as helpful, and generated images outperforming baseline methods in quality.
This work marks a step forward in making AI more interactive, interpretable, and user-centric. By enabling models to actively understand and clarify, it opens new avenues for AI-assisted art, design, and beyond. Future work will explore multimodal belief representations, reinforcement learning for question optimization, and real-world user testing, aiming to create truly intelligent, collaborative AI systems.
Deep Analysis
Background
随着生成式AI的快速发展，文本到图像（T2I）技术已取得显著突破，如DALL·E、Stable Diffusion等模型实现了高质量图像生成。然而，提示不充分或模糊仍是主要瓶颈，导致生成结果偏离用户预期。过去的研究多关注模型优化、数据增强，缺乏主动交互机制，难以满足多样化需求。符号推理和场景图等符号化表示逐步引入，但多轮交互和不确定性表达仍待突破。近年来，结合大模型的交互系统逐渐出现，但缺乏系统性框架支持多轮信念表达与动态提问，限制了实际应用。
Core Problem
当前T2I模型多为被动响应式，难以理解用户深层意图，导致反复调试。提示不充分或模糊时，模型难以自主识别关键元素，影响生成质量。用户体验差，尤其在艺术和设计场景中，个性化需求难以满足。缺乏有效的交互策略限制了模型的智能化水平。解决这一问题需要模型主动识别不确定性，动态与用户沟通，优化交互流程，提升效率和满意度。
Innovation
本研究的核心创新包括：1）引入符号化的信念图，表达实体、属性和关系的不确定性，增强模型的可解释性；2）结合大规模语言模型（如GPT-4）实现主动提问，自动识别关键不确定元素；3）设计多阶段问答策略，优化提问选择，减少交互轮次；4）建立自动化评估流程，模拟用户意图，结合图像相似度指标验证效果。这一框架突破了传统被动模型的局限，为多模态交互提供新思路。
Methodology
- �� 构建信念图：解析用户提示，识别实体、属性、关系，赋予概率和重要性评分。
- �� 多轮交互：模型根据信念图识别最不确定且重要的元素，利用LLM生成澄清问题。
- �� 提问策略：结合启发式打分和信念图信息，生成多项选择或开放式问题，确保简洁易答。
- �� 信念更新：用户回答后，模型更新信念图，调整概率和重要性。
- �� 图像生成：用更新后的提示调用T2I模型，生成符合用户意图的图像。
- �� 评估机制：模拟用户意图，结合VQAScore等指标，自动评估生成效果。
- �� 模块化设计：各组件可独立优化，支持不同模型和策略的集成。
Experiments
采用ImageInWords、COCO和DesignBench三大数据集，模拟用户意图，比较单轮与多轮交互效果。指标包括VQAScore、图像相似度和交互轮次。对比基线如DALL·E，进行ablation分析验证问答策略效果。人类评估补充，确保系统在艺术设计中的实用性。调优超参数如最大轮次和重要性阈值，平衡交互成本与生成质量。
Results
实验显示，代理在5轮内实现VQAScore超过0.8，比单轮提升2倍以上。主动提问显著减少用户调试时间，提升图像符合度。人类评估表明，90%以上受试者认为信念图和问答机制有助于表达意图，生成图像优于传统方法。多轮交互有效缓解提示不足问题，提升模型理解能力。自动模拟验证显示问答策略鲁棒性强，适应多样场景。
Applications
该方法适用于艺术创作、产品设计、虚拟场景构建等领域，用户可通过多轮交互实现个性化定制。系统可集成于图像编辑软件、虚拟现实平台，降低专业门槛，提升创作效率。未来还可结合用户偏好模型，实现更智能的个性化推荐和自动优化。
Limitations & Outlook
信念图构建依赖LLM的准确性，存在偏差风险。在复杂场景中，问答策略可能冗余或遗漏关键信息。交互过程计算成本较高，需优化模型效率。实际应用中，用户行为多样，系统需适应不同偏好和习惯，未来需增强鲁棒性和个性化能力。
Plain Language Accessible to non-experts
想象你在厨房做菜，但只告诉厨师“做个好吃的菜”。厨师不知道你喜欢的口味或偏好食材，可能会做出不合你心意的菜。于是，厨师开始问你：“喜欢辣吗？”“要不要加点葱？”每次你回答后，他会根据你的偏好调整食谱，直到你满意。这就像这个研究中的AI助手，它通过主动问问题，逐步了解你真正想要的图片。它用一种“信念图”记录自己对场景的理解和不确定性，就像厨师记笔记一样。这样，AI不再被动接受指令，而变成了一个善于沟通、懂得提问的“厨师”，帮你做出更满意的作品。
ELI14 Explained like you're 14
你知道有时候你想让朋友帮你画画，但只说了“画个漂亮的风景”，朋友可能不知道你喜欢什么样的风景。有时候他们会问：“喜欢山吗？”“喜欢海吗？”这样他们就能画得更符合你的心意。这就像这个研究的AI助手，它会主动问你一些问题，帮你更清楚你想要的画面。它用一种叫“信念图”的东西，把你说的内容和它自己猜测的不确定部分都记下来。每次你回答问题，它就会更新自己的想法，然后再画出更贴近你心意的图片。这样，画画变得更有趣，也更容易得到你喜欢的作品。
Abstract
User prompts for generative AI models are often underspecified, leading to a misalignment between the user intent and models' understanding. As a result, users commonly have to painstakingly refine their prompts. We study this alignment problem in text-to-image (T2I) generation and propose a prototype for proactive T2I agents equipped with an interface to (1) actively ask clarification questions when uncertain, and (2) present their uncertainty about user intent as an understandable and editable belief graph. We build simple prototypes for such agents and propose a new scalable and automated evaluation approach using two agents, one with a ground truth intent (an image) while the other tries to ask as few questions as possible to align with the ground truth. We experiment over three image-text datasets: ImageInWords (Garg et al., 2024), COCO (Lin et al., 2014) and DesignBench, a benchmark we curated with strong artistic and design elements. Experiments over the three datasets demonstrate the proposed T2I agents' ability to ask informative questions and elicit crucial information to achieve successful alignment with at least 2 times higher VQAScore (Lin et al., 2024) than the standard T2I generation. Moreover, we conducted human studies and observed that at least 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow, highlighting the effectiveness of our approach. Code and DesignBench can be found at https://github.com/google-deepmind/proactive_t2i_agents.