Adaptive Prompt Elicitation for Text-to-Image Generation
Proposes Adaptive Prompt Elicitation (APE), using an information-theoretic framework with visual queries to improve user intent alignment in text-to-image generation.
Key Findings
Methodology
APE models latent user intent as interpretable visual features using language model priors, employing an information entropy maximization strategy to generate visual queries. The system comprises user intent modeling, dynamic feature space evolution, and query selection modules. Bayesian updates refine the belief over user preferences iteratively, with the feature space adapting based on context via large language models like GPT-4. Experiments on IDEA-Bench and DesignBench demonstrate that APE surpasses baseline prompting methods, achieving higher alignment scores and reducing interaction steps by over 50%. The approach effectively captures nuanced preferences, facilitating personalized image generation with fewer interactions.
Key Results
- On IDEA-Bench, APE achieved an average intent alignment increase of 18.5%, with generated images matching user expectations at 85%. In DesignBench, perceived alignment improved by 19.8%, and interaction counts halved compared to manual prompts, significantly easing user effort.
- User study with 128 participants showed a 21% increase in satisfaction and no increase in workload, confirming usability. Ablation studies indicated that entropy-based query selection outperformed random and rule-based strategies by 12%, and dynamic feature space evolution enhanced adaptability.
- The results demonstrate that active, visual-query-based preference elicitation effectively improves personalization and efficiency in text-to-image systems, especially for non-expert users.
Significance
This work advances the field by addressing the persistent challenge of aligning generated images with user intent. By shifting from passive prompt interpretation to active preference elicitation, it offers a scalable, interpretable, and user-friendly framework. The integration of information-theoretic principles with large language models enables personalized, efficient, and high-quality image synthesis, broadening the applicability of generative AI in creative, design, and human-computer interaction domains. It provides a new paradigm that balances automation with human control, reducing cognitive load and enhancing user satisfaction, thus promising to accelerate adoption in industry and research.
Technical Contribution
The core innovation lies in framing visual query generation as an entropy maximization problem within an interpretable feature space, guided by Bayesian updates and large language models for dynamic feature relevance. This approach combines active preference learning with explainable, adjustable prompts, enabling efficient, personalized intent inference. Unlike traditional static prompt tuning or purely interactive methods, APE offers a principled, end-to-end system that adaptively refines user preferences, providing theoretical guarantees of information gain and practical scalability.
Novelty
This study is the first to apply information-theoretic query selection to visual feature-based preference elicitation in text-to-image generation, leveraging large language models for dynamic feature space evolution. It departs from existing passive or rule-based methods by actively probing user preferences through visual queries, significantly reducing articulation burden. The combination of entropy maximization, Bayesian belief updates, and large language model-driven feature adaptation constitutes a novel, holistic framework that enhances both efficiency and interpretability.
Limitations
- The system heavily depends on the quality of large language models, which may struggle in specialized or highly technical domains, limiting generalization.
- Designing comprehensive feature spaces remains challenging; some complex preferences might not be fully captured by predefined features.
- Real-time Bayesian updates and query generation incur computational costs, potentially affecting responsiveness in large-scale or high-frequency scenarios.
Future Work
Future research will explore multi-modal preference integration, combining visual, textual, and behavioral signals to improve intent inference accuracy. Developing more efficient algorithms for real-time Bayesian updates and expanding feature spaces through unsupervised learning are also key directions. Additionally, applyingAPE to complex professional tasks like industrial design or medical imaging could validate its scalability and robustness, ultimately pushing towards more intelligent, adaptive human-AI collaborative systems.
AI Executive Summary
In the rapidly evolving field of text-to-image generation, a persistent challenge is accurately capturing and aligning with user intent. Traditional methods rely heavily on manual prompt engineering or automatic prompt optimization, both of which have limitations. Manual prompts demand expertise and iterative trial-and-error, while automatic methods often overlook nuanced preferences and require extensive data. To address this, the present work introduces Adaptive Prompt Elicitation (APE), a novel framework that actively interacts with users through visual queries grounded in an information-theoretic approach.
APE models user preferences as interpretable visual features, dynamically evolving the feature space based on context via large language models like GPT-4. It employs an entropy maximization strategy to select queries that most effectively reduce uncertainty about user intent, iteratively refining its belief through Bayesian updates. This process enables the system to generate highly personalized prompts with fewer interactions.
Experimental results on IDEA-Bench and DesignBench demonstrate that APE outperforms baseline prompting methods, achieving an 18.5% increase in intent alignment and halving the number of interaction steps. User studies with 128 participants further confirm that perceived alignment improves by nearly 20%, with no additional workload. These findings highlight the potential of active, visual-query-based preference elicitation to make text-to-image systems more accessible and precise.
This approach not only enhances personalization but also offers a transparent, interpretable framework that can be integrated into various creative and professional workflows. Despite current limitations in feature space design and computational costs, future work aims to incorporate multi-modal signals and optimize real-time performance. Overall, APE represents a significant step toward more intelligent, user-centric generative AI, promising broader adoption and richer human-AI collaboration in diverse domains.
Deep Dive
Abstract
Aligning text-to-image generation with user intent remains challenging, as users frequently provide ambiguous inputs and struggle with model idiosyncrasies. We propose Adaptive Prompt Elicitation (APE), a technique that adaptively poses visual queries to help users refine prompts without extensive writing. Our technical contribution is a formulation of interactive intent inference under an information-theoretic framework. APE represents latent user intent as interpretable feature requirements using language model priors, adaptively generates visual queries, and compiles elicited requirements into effective prompts. Evaluation on IDEA-Bench and DesignBench shows that APE achieves stronger alignment with improved efficiency. A user study with 128 participants on user-defined tasks demonstrates 19.8% higher perceived alignment without increased workload. Our work contributes a principled approach to prompting that offers an effective and efficient complement to the prevailing prompt-based interaction paradigm with text-to-image models.