GenEvolve: Self-Evolving Image Generation Agents via Tool-Orchestrated Visual Experience Distillation
GenEvolve achieves self-evolving image generation via tool-orchestrated visual experience distillation, enhancing generation quality.
Key Findings
Methodology
GenEvolve employs a tool-orchestrated visual experience distillation framework, modeling each generation attempt as a tool-orchestrated trajectory. The agent gathers evidence, selects references, invokes generation skills, and composes them into a prompt-reference program. This method contrasts multiple trajectories for the same request, abstracting best-worst differences into structured visual experience provided only to a privileged teacher branch.
Key Results
- On GENEVOLVE-BENCH, GenEvolve outperforms existing benchmarks in KScore and four judge dimensions, showing significant performance improvement.
- In the WISE benchmark, GenEvolve consistently surpasses SOTA direct generators and recent agentic baselines.
- Through visual experience distillation, the student model exhibits better search, knowledge activation, reference selection, and prompt construction.
Significance
GenEvolve is significant in the field of image generation, addressing limitations of existing methods under diverse and complex requests. By tool orchestration and visual experience distillation, it enhances generation quality and adaptability, advancing image generation technology.
Technical Contribution
GenEvolve's technical contributions include redefining open-ended image generation as an agentic trajectory learning problem. By tool orchestration and visual experience distillation, it optimizes tool use, reference selection, knowledge activation, and prompt-reference program construction.
Novelty
GenEvolve is the first to combine tool orchestration and visual experience distillation for self-evolving image generation agents. It innovates in tool use and experience internalization compared to existing methods.
Limitations
- May perform poorly with extremely complex requests due to the need for more intricate tool orchestration.
- Dependency on tools and references may lead to performance degradation in resource-scarce environments.
Future Work
Future research directions include expanding tool integration capabilities, enhancing adaptability to complex requests, and exploring more visual experience distillation strategies.
AI Executive Summary
GenEvolve redefines open-ended image generation problems through a tool-orchestrated visual experience distillation framework. Traditional image generation methods often rely on a single prompt-to-image conversion, whereas GenEvolve enhances generation quality and adaptability through multi-trajectory comparison and visual experience distillation.
In experiments, GenEvolve excels in GENEVOLVE-BENCH and WISE benchmarks, surpassing existing image generation frameworks. Its core technologies include tool orchestration and visual experience distillation, optimizing tool use, reference selection, and knowledge activation through trajectory comparisons.
Despite significant advancements in image generation, GenEvolve faces challenges in handling extremely complex requests. Future research will focus on expanding tool integration capabilities and exploring more visual experience distillation strategies to further enhance generation performance.
Deep Analysis
Background
Recent years have seen significant advances in image generation technology, yet challenges remain in handling open-ended and complex requests. Traditional methods often rely on a single prompt-to-image conversion, struggling to meet diverse generation demands. GenEvolve offers a new solution through tool orchestration and visual experience distillation.
Core Problem
The core of open-ended image generation lies in effectively coordinating internal generative capabilities with external resources. Existing methods often struggle to fully leverage external references and generative knowledge when handling diverse and complex requests.
Innovation
GenEvolve's core innovation lies in modeling each generation attempt as a tool-orchestrated trajectory and optimizing the generation process through visual experience distillation. This approach not only improves generation quality but also enhances system adaptability.
Methodology
- �� Model each generation attempt as a tool-orchestrated trajectory
- �� Agent gathers evidence, selects references, invokes generation skills
- �� Compose elements into a prompt-reference program
- �� Optimize student model through visual experience distillation
Experiments
Experiments were conducted on GENEVOLVE-BENCH and WISE benchmarks, using Nano Banana Pro and Qwen-Image-Edit as downstream generators. Evaluation metrics include KScore and four judge dimensions.
Results
Results show that GenEvolve surpasses existing benchmarks on GENEVOLVE-BENCH and WISE, particularly excelling under complex requests.
Applications
GenEvolve can be applied in fields like architectural design, scientific illustration, and creative transfer, especially suitable for scenarios requiring high-quality and complex generation.
Limitations & Outlook
While GenEvolve excels in generation quality, challenges remain in handling extremely complex requests. Additionally, dependency on external tools and references may limit its application in resource-scarce environments.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. GenEvolve is like a smart chef assistant that not only follows recipes but also adjusts ingredients based on your taste. It first looks up ingredient information, selects suitable spices, and then combines culinary skills to create a delicious dish. This process is akin to GenEvolve optimizing the generation process through tool orchestration and visual experience distillation, enhancing image quality.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super cool game with an assistant helping you design game characters. This assistant is like GenEvolve; it doesn't just draw characters based on your description but also looks up info, selects reference images, and uses its skills to optimize the design. This way, your character becomes super awesome!
Glossary
GenEvolve
A framework for self-evolving image generation through tool orchestration and visual experience distillation.
Used to enhance image generation quality and adaptability.
Visual Experience Distillation
Abstracts best-worst trajectory differences into visual experience through trajectory comparison.
Used to optimize the student model's generation process.
Tool Orchestration
Effectively utilizing external tools and reference resources during the generation process.
Helps agents improve generation quality under complex requests.
Trajectory Learning
Models the generation process as a series of tool calls and decisions.
Key technology for optimizing the generation process.
Nano Banana Pro
A high-quality image generation and editing tool.
Used as a downstream generator in experiments.
Open Questions Unanswered questions from this research
- 1 How to optimize tool orchestration and reference selection in resource-scarce environments?
- 2 How to further enhance generation quality under complex requests?
Applications
Immediate Applications
Architectural Design
GenEvolve can generate complex architectural design images, aiding designers in better realizing creative concepts.
Long-term Vision
Intelligent Creative Assistant
In the future, GenEvolve could evolve into an intelligent creative assistant, supporting creative generation and optimization across multiple domains.
Abstract
Open-ended image generation is no longer a simple prompt-to-image problem. High-quality generation often requires an agent to combine a model's internal generative ability with external resources. As requests become more diverse and demanding, we aim to develop a general image-generation agent that can self-evolve through trajectories and use tools more effectively across varied generation challenges. To this end, we propose GenEvolve, a self-evolving framework based on Tool-Orchestrated Visual Experience Distillation. In GenEvolve, each generation attempt is modeled as a tool-orchestrated trajectory, where the agent gathers evidence, selects references, invokes generation skills, and composes them into a prompt-reference program. Unlike existing agentic generation methods that mainly rely on image-level scalar rewards, GenEvolve compares multiple trajectories for the same request and abstracts best-worst differences into structured visual experience, provided only to a privileged teacher branch. Inspired by on-policy self-distillation, Visual Experience Distillation provides dense token-level supervision, helping the student internalize better search, knowledge activation, reference selection, and prompt construction. We further construct GenEvolve-Data and GenEvolve-Bench. Experiments on public benchmarks and GenEvolve-Bench show substantial gains over strong baselines, achieving state-of-the-art performance among current image-generation frameworks. Our website is as follows: https://ephemeral182.github.io/GenEvolve/