Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence
The 'Agentic Self-Improvement' framework employs a two-stage goal-driven optimization, significantly improving semantic adherence and control in image-to-video generation, outperforming unguided methods with up to 69% preference.
Key Findings
Methodology
This framework transforms video synthesis into a closed-loop, goal-oriented optimization process, comprising two main stages. The first stage employs a multimodal large language model (mLLM) to generate structured questions—Davidsonian Scene Graphs (DSG) and Common Mistake Questions (CMQ)—which evaluate semantic fidelity and artifact presence. The mLLM iteratively refines the input prompt based on automated question-answering feedback, maximizing semantic alignment scores such as the Video-Text Adherence (VTA). The second stage applies Bayesian optimization to jointly tune stochastic seeds and CFG scales, guided by a multi-metric reward function including UVQ, RAHF, and VTA. This systematic approach ensures high-quality, controllable video outputs.
Key Results
- User preference studies show videos generated via the agentic framework are preferred up to 69%, a substantial improvement over unguided search (~20-30%).
- Prompt refinement improved human preference scores by 27%, with statistical significance (p<0.001).
- Bayesian hyperparameter tuning over 100 iterations increased UVQ, RAHF, and VTA scores by over 20% on average, validating the efficiency of goal-directed search.
Significance
This work addresses the longstanding challenge of controlling black-box diffusion models for professional video synthesis. By integrating structured semantic evaluation and Bayesian parameter tuning, it enhances the reliability, stability, and predictability of generated videos. The approach bridges the gap between raw fidelity and precise control, facilitating deployment in content creation, virtual production, and entertainment industries. Its modular design allows adaptation across diverse multimodal generation tasks, marking a significant step toward production-ready AI-driven content pipelines.
Technical Contribution
The paper introduces the 'Agentic Self-Improvement' framework, combining structured semantic evaluation via DSG and CMQ, automated prompt refinement with a multimodal LLM, and Bayesian optimization for hyperparameters. The Video-Text Adherence (VTA) metric quantifies content alignment based on hierarchical question-answering, enabling precise feedback-driven optimization. This integration of multimodal understanding, structured evaluation, and probabilistic parameter search constitutes a novel methodology for controllable, high-fidelity video synthesis from black-box models.
Novelty
This is the first comprehensive system that employs multimodal LLMs for prompt iteration, structured semantic and artifact evaluation, and Bayesian joint tuning of stochastic and deterministic parameters in image-to-video generation. Unlike prior works limited to raw fidelity or manual tuning, this approach introduces an automated, goal-oriented pipeline that significantly improves semantic adherence and stability, representing a new paradigm in controllable content synthesis.
Limitations
- The effectiveness heavily depends on the quality of DSG and CMQ question generation; inaccuracies here can mislead the optimization process.
- Bayesian optimization, while efficient, still faces scalability issues in very high-dimensional parameter spaces or highly complex models.
- The evaluation metrics, though comprehensive, are primarily static and may not fully capture dynamic temporal nuances or subjective aesthetic preferences. Future work should incorporate adaptive, real-time feedback mechanisms.
Future Work
Future directions include enhancing the robustness of structured question generation, integrating reinforcement learning for dynamic control, and extending the framework to multi-task scenarios involving style, motion, and content editing. Additionally, developing real-time optimization and feedback systems could further improve practical deployment. Exploring broader multimodal datasets and more sophisticated evaluation metrics will also be crucial to generalize the approach across diverse content domains.
AI Executive Summary
The rapid advancement of black-box diffusion models has revolutionized automated content creation, enabling high-fidelity image and video synthesis from textual prompts. However, their inherent stochasticity and lack of fine-grained control pose significant barriers to professional adoption. Minor variations in hyperparameters or prompts often lead to unpredictable outputs, making reliable content generation a challenge. This has created a critical need for systematic, goal-oriented control mechanisms that can steer generative models toward desired outcomes with minimal trial-and-error.
Addressing this challenge, the authors introduce the 'Agentic Self-Improvement' framework, a novel methodology that transforms the video synthesis pipeline into a closed-loop, goal-driven optimization system. The core idea is to leverage structured semantic evaluation and probabilistic parameter tuning to enhance controllability and output quality. The framework comprises two main stages: first, an iterative prompt refinement process employs a multimodal large language model (mLLM) to generate and answer structured questions—Davidsonian Scene Graphs (DSG) and Common Mistake Questions (CMQ)—which evaluate the semantic fidelity and artifact presence in generated videos. This automated question-answering loop guides prompt adjustments, maximizing a semantic alignment score called Video-Text Adherence (VTA).
The second stage involves Bayesian optimization, which efficiently searches the hyperparameter space—including stochastic seeds and CFG scales—guided by a multi-metric reward function that combines perceptual quality (UVQ, RAHF) and semantic adherence (VTA). This joint optimization ensures that the final outputs are both visually high-quality and semantically aligned with the input prompts. Extensive experiments on the V-Bench dataset demonstrate that videos generated through this framework outperform baseline unguided methods, with preference rates reaching up to 69%. Human preference studies confirm the perceptual improvements, while quantitative metrics show consistent gains across multiple quality indicators.
This work significantly advances the controllability and reliability of black-box diffusion models, opening pathways for their deployment in professional content creation, virtual production, and interactive media. Its modular design allows adaptation to various multimodal generation tasks, fostering further research into goal-oriented, autonomous content synthesis. Despite current limitations related to question generation accuracy and computational costs, the proposed framework sets a new standard for systematic, automated control in generative AI, promising a future where high-quality, semantically faithful videos are produced reliably and efficiently.
Deep Analysis
Background
近年来，深度学习推动了图像到视频生成技术的飞跃，从早期的GANs和VAEs到后来的扩散模型（如Veo系列）实现了高分辨率、时间一致的生成。早期方法在时间连续性和语义保持方面存在不足，难以满足专业内容制作的需求。Diffusion Transformer（DiT）等大模型的出现，极大提升了生成的真实性和细节还原能力，但其黑箱特性限制了控制性和可重复性。行业内逐渐认识到，单纯依赖模型的原始输出难以满足内容创作的高标准，亟需引入结构化语义检测和目标导向的调节机制，以实现更精细的内容控制。
Core Problem
现有的黑箱图像到视频模型在内容控制方面存在明显瓶颈。微小的超参数调整（如随机种子、CFG尺度）会导致输出结果的剧烈变化，严重影响模型的稳定性和可控性。这限制了模型在专业应用中的推广，尤其是在需要严格语义一致性和Artifact控制的场景中。传统的试错调节方式既低效又昂贵，难以满足实际生产需求。因此，如何设计一种系统化的自动调节框架，实现内容的高精度控制，成为亟待解决的问题。
Innovation
本研究的创新点主要包括：1）提出“Agentic Self-Improvement”框架，将视频生成转化为闭环、目标导向的优化任务，突破传统单向生成的限制；2）利用多模态大语言模型（如Gemini）自动生成结构化问答（DSG和CMQ），实现提示的自动迭代优化，确保语义一致性和Artifact检测；3）引入基于结构化问答的VTA指标，量化视频内容与文本提示的匹配程度，为生成内容提供明确的反馈依据；4）采用贝叶斯优化联合调节随机种子和CFG尺度，提升参数调节效率，减少试错成本。这些创新融合了多模态理解、结构化语义检测和贝叶斯决策，为黑箱模型的可控性提供了新思路。
Methodology
- �� 输入：用户提供的文本提示和初始图像。
- �� 第一步：利用多模态大语言模型（如Gemini）生成一组结构化问答（DSG和CMQ），对场景语义和Artifact进行检测。
- �� 第二步：基于生成的视频，自动化VQA系统（Gemini 2.5 Pro）回答问答，评估内容一致性和Artifact。
- �� 第三步：根据问答结果，模型自动调整提示，进行多轮迭代，最大化内容符合度（如VTA得分）。
- �� 第四步：在优化完成后，利用贝叶斯优化（如Google Vizier）调节超参数（随机种子和CFG尺度），以最大化多指标（UVQ、RAHF、VTA）。
- �� 输入：优化后的提示和超参数。
- �� 第五步：生成最终视频，进行多指标评估，确保内容质量和控制效果。
Experiments
采用V-Bench数据集进行验证，比较基线无指导方法和本框架的性能。通过人类偏好测试，评估生成视频的真实感和语义一致性。设置不同的优化轮次（10次和100次），观察偏好率变化。使用自动化指标（UVQ、RAHF、VTA）进行定量分析，验证优化效果。AB测试中，优化提示后偏好提升27%，贝叶斯调优在多指标上平均提升20%以上。实验还包括不同场景和复杂度的测试，验证框架的泛化能力。
Results
实验显示，基于Agentic框架的视频偏好率最高达69%，远超随机搜索（20-30%）。提示优化模块显著改善了内容一致性，偏好提升27%。贝叶斯调优在100次迭代中，UVQ、RAHF、VTA指标平均提升20%以上，验证了目标导向调节的有效性。多指标融合的评价体系，能更全面反映生成内容的质量，显著优于单一指标评估。整体结果表明，该方法在提升内容控制和稳定性方面具有巨大潜力。
Applications
该框架适用于影视制作、广告创意、虚拟现实等行业，尤其在需要高语义一致性和内容控制的场景中。用户只需提供文本提示和输入图像，系统即可自动优化提示和超参数，生成符合预期的高质量视频。未来，可结合实时反馈机制，应用于交互式内容创作和自动化生产线，极大提升内容生产效率和质量一致性。长远来看，该技术有望推动虚拟主播、数字孪生等新兴产业的发展，实现智能化、个性化的内容生成。
Limitations & Outlook
当前方法对结构化问答生成的依赖较大，问答体系的准确性直接影响优化效果。在极端复杂或多模态场景中，贝叶斯调节仍面临维度灾难，调优效率有限。此外，指标体系主要基于静态评价，难以完全捕捉动态视频中的细节变化。未来需要引入更鲁棒的多模态理解和自适应评价机制，以应对多样化应用需求。模型训练和调优过程计算成本较高，限制了实时应用的可能性。
Plain Language Accessible to non-experts
想象你在厨房里做菜。每次你想做一道新菜，都需要调整火候、调料的用量和烹饪时间，才能做出满意的味道。传统的方法就像是试错：多次尝试，直到找到合适的配比。而这篇论文提出了一个聪明的厨师助手，它能根据你之前的尝试，自动分析哪些调料用得不对，哪些火候不够，然后给出改进建议。接着，它还会根据你喜欢的口味，调整烹饪的时间和火力，确保每次做出来的菜都符合你的期待。这个助手用了一种叫“目标导向优化”的方法，结合了“问答检测”和“智能调节”，让厨房变得更高效、更精准。最终，你不用再反复试错，就能轻松做出美味佳肴。这就像是给厨房装上了一个聪明的机器人助手，帮你把复杂的调味和火候调节变得简单又可靠。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的电子游戏。每次你想赢，都得调节很多参数，比如角色的装备、技能的使用顺序、甚至每次跳跃的力度。可是，这个游戏很难控制，稍微调整一下参数，结果就可能完全不一样。有时候你试了很多次，花了很多时间，才能找到最好的组合。这篇论文就像是发明了一台智能机器人，它可以帮你自动调节这些参数，让你更快找到最好的策略。它会观察你每次的表现，问自己：“这个角色的动作是不是符合游戏规则？装备是不是合理？”然后自己调整参数，再试一次。它还会用一种叫贝叶斯的方法，聪明地选择哪些参数值得尝试，避免浪费时间。最终，你可以用更少的尝试，赢得比赛，感觉就像有了一个超级聪明的助手在帮你。这个方法让复杂的调节变得简单又高效，就像有个机器人帮你打理一切一样。
Abstract
Modern black-box Image-to-Video (I2V) models offer powerful capabilities in automated content creation, yet their lack of fine-grained control and reliability presents significant challenges in professional workflows. Their inherent stochasticity causes minor variations in textual prompts or hyperparameters to yield drastically different outputs often necessitating inefficient, brute-force trial-and-error processes. To address these limitations, we introduce the ``Agentic Self-Improvement" framework, which reframes video synthesis into a closed-loop, goal-directed optimization. Our framework systematically navigates the generation parameter space using a novel two-stage approach. In the first stage, an iterative prompt optimization loop uses a multimodal Large Language Model (mLLM) to refine the input prompt. This refinement implements two automated evaluations: Davidsonian Scene Graph (DSG) queries ensure semantic adherence, and Common Mistake Questions (CMQ) for artifact detection. At the second stage, we use Bayesian optimization to efficiently co-optimize stochastic seeds and CFG scales. This search is guided by a suite of quality metrics, including the novel Video-Text Adherence (VTA) score derived from the DSG and CMQ evaluations. Our framework significantly outperforms unguided search methods: in human preference studies, videos generated via our agentic approach were strongly preferred over baseline outputs, achieving win rates up to 69\%. This work provides a practical and extensible methodology for enhancing the predictability and control of state-of-the-art video generation models, moving the field beyond speculative curiosities toward reliable, production-ready tools.
References (20)
VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models
Ziqi Huang, Fan Zhang, Xiaojie Xu et al.
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
Jaemin Cho, Yushi Hu, Roopal Garg et al.
GENERATIVE ADVERSARIAL NETS
Individualized Treat, Jinsung Yoon
Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs
Zicheng Zhang, Ziheng Jia, Haoning Wu et al.
Rich Human Feedback for Text-to-Image Generation
Youwei Liang, Junfeng He, Gang Li et al.
VPO: Aligning Text-to-Video Generation Models with Prompt Optimization
Jiale Cheng, Ruiliang Lyu, Xiaotao Gu et al.
WorldSimBench: Towards Video Generation Models as World Simulators
Yiran Qin, Zhelun Shi, Jiwen Yu et al.
Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance
Dazhong Shen, Guanglu Song, Zeyue Xue et al.
Video-to-Video Synthesis
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu et al.
The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-To-Video Generation
Bingjie Gao, Xinyu Gao, Xiaoxue Wu et al.
VideoMAR: Autoregressive Video Generatio with Continuous Tokens
Hu Yu, Biao Gong, Hangjie Yuan et al.
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.
Training Diffusion Models with Reinforcement Learning
Kevin Black, Michael Janner, Yilun Du et al.
Rich features for perceptual quality assessment of UGC videos
Yilin Wang, Junjie Ke, Hossein Talebi et al.
Make It Move: Controllable Image-to-Video Generation with Text Descriptions
Yaosi Hu, Chong Luo, Zhenzhong Chen
Generating Videos with Scene Dynamics
Carl Vondrick, H. Pirsiavash, A. Torralba
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
A. Blattmann, Robin Rombach, Huan Ling et al.
Prompt-A-Video: Prompt your Video Diffusion Model via Preference-Aligned LLM
Yatai Ji, Jiacheng Zhang, Jie Wu et al.