Improving Video Generation with Human Feedback
This work introduces human-preference-guided reinforcement learning for video generation, leveraging VideoReward to improve flow-based models, achieving significant quality gains.
Key Findings
Methodology
The study constructs a large-scale human preference dataset with 182k annotated triplets across visual quality, motion smoothness, and text alignment, using 12 modern T2V models. A multi-dimensional reward model, VideoReward, is trained with a VLM backbone (Qwen2-VL-2B). Based on this, three alignment algorithms—Flow-DPO, Flow-RWR, and Flow-NRG—are developed, integrating reinforcement learning principles with KL regularization. Extensive experiments demonstrate that VideoReward surpasses existing reward models, with Flow-DPO achieving superior preference alignment and Flow-NRG enabling personalized inference adjustments.
Key Results
- VideoReward outperforms prior reward models, with preference accuracy reaching 72.89% overall and 73.59% on modern T2V models, representing over 20% improvement. Flow-DPO, especially with a fixed KL coefficient, consistently yields better alignment across multiple preference dimensions, reducing reward hacking and artifacts. Flow-NRG allows users to dynamically weight multiple objectives during inference, enabling personalized video quality control.
- In comparative evaluations, VideoReward demonstrates robust generalization across datasets, outperforming models like VideoScore, LiFT, and VisionReward. Ablation studies confirm that the multi-dimensional approach and stable KL regularization are crucial for reliable preference modeling. The combined framework significantly enhances motion smoothness, content fidelity, and text alignment in generated videos.
- The inference-time Flow-NRG method offers flexible, user-controlled customization, making it suitable for personalized content creation in entertainment, virtual reality, and social media applications. The approach also opens pathways for adaptive multi-objective optimization in generative modeling.
Significance
This research addresses longstanding challenges in aligning flow-based video generation models with human preferences, bridging the gap between high-fidelity synthesis and user-centric quality. By integrating large-scale preference data, sophisticated reward modeling, and novel alignment algorithms, it advances the state-of-the-art in personalized, high-quality video synthesis. The framework enhances the practical deployment of AI-generated videos in entertainment, advertising, and immersive media, fostering more natural, engaging content. It also sets a foundation for future research in multi-objective reinforcement learning and adaptive content generation, with potential to transform how AI interacts with human aesthetic and functional expectations.
Technical Contribution
The paper introduces a comprehensive multi-dimensional preference dataset and a robust VideoReward model, systematically analyzing design choices. It innovates with Flow-DPO, a stable, time-independent training strategy, and Flow-NRG, a flexible inference-time guidance method that allows dynamic multi-objective weighting. These contributions extend RLHF techniques to flow-based models, providing theoretical guarantees and practical tools for preference alignment. The combination of large-scale data, reward modeling, and adaptive inference constitutes a significant methodological advancement in AI-driven video synthesis.
Novelty
This work is the first to systematically incorporate multi-dimensional human preferences into flow-based video generation via RLHF. It introduces Flow-DPO, which removes the timestep-dependent KL regularization problematic in diffusion models, and Flow-NRG, enabling real-time, user-controlled multi-objective adjustment during inference. These innovations outperform traditional fine-tuning and single-metric optimization, filling a critical gap in personalized, high-quality video synthesis research.
Limitations
- The approach relies heavily on large-scale annotated preference data, which is costly and time-consuming to acquire, potentially limiting scalability. Biases in human preferences may affect model fairness and robustness.
- Despite improvements, the models still struggle with complex scenes and long videos, where motion artifacts and detail loss can occur. Further architectural enhancements are needed.
- The inference-time adjustment, while flexible, requires manual tuning of multiple weights, lacking an automated optimization mechanism, which may hinder practical deployment in real-time applications.
Future Work
Future research will focus on reducing dependence on extensive human annotations, possibly through semi-supervised or self-supervised methods. Developing automatic multi-objective weight tuning algorithms will enhance usability. Extending the framework to longer, more complex videos and integrating multimodal preferences could further improve realism and personalization, broadening AI’s role in creative industries.
AI Executive Summary
Video generation technology has rapidly advanced with the advent of diffusion and flow-based models, enabling the creation of highly realistic and coherent videos. However, current systems still face significant challenges, including unnatural motion, misalignment with prompts, and limited ability to incorporate human preferences. Traditional training approaches often optimize for single metrics or rely on limited datasets, which do not fully capture the nuanced preferences of users.
This paper addresses these issues by developing a comprehensive reinforcement learning framework guided by human preferences. Central to this approach is the construction of a large-scale, multi-dimensional preference dataset, comprising 182,000 triplets annotated across visual quality, motion smoothness, and text alignment. Using this dataset, the authors train VideoReward, a multi-dimensional reward model based on a vision-language backbone (Qwen2-VL-2B). This model effectively captures human preferences, providing a reliable signal for optimizing video generation.
Building on this, the authors propose three novel algorithms: Flow-DPO, Flow-RWR, and Flow-NRG. Flow-DPO is a training strategy that aligns flow-based models with human preferences by removing timestep-dependent KL regularization, ensuring stability and robustness. Flow-RWR employs reward-weighted regression to fine-tune models, while Flow-NRG offers an inference-time guidance mechanism that allows users to dynamically adjust multiple objectives, enabling personalized video outputs.
Experimental results demonstrate that their methods outperform existing reward models and alignment strategies. VideoReward achieves over 70% preference accuracy, significantly surpassing baselines. Flow-DPO consistently yields better multi-dimensional alignment, avoiding reward hacking and artifacts. Flow-NRG enables real-time, user-controlled customization, making the system adaptable to diverse user needs.
This work marks a significant step forward in aligning flow-based video models with human preferences, combining large-scale data, robust reward modeling, and flexible inference techniques. It opens new avenues for personalized, high-quality video synthesis, with broad applications in entertainment, virtual reality, and content creation. Future efforts will aim to reduce data dependency, automate multi-objective tuning, and extend capabilities to more complex, longer videos, further pushing the boundaries of AI-generated media.
Deep Analysis
Background
视频生成技术经历了从早期基于GAN、VAE的短视频合成，到近年来采用扩散模型和流式模型实现高保真长视频。代表性工作包括DALL·E、Imagen、Stable Diffusion在图像领域的突破，以及VideoDiffusion、Make-A-Video等在视频领域的创新。尽管如此，现有模型在运动平滑、内容一致性和用户偏好满足方面仍有限。强化学习人类反馈（RLHF）在文本和图像生成中已取得成功，但在视频生成中仍处于探索阶段，主要受限于偏好数据不足和奖励模型设计不完善。
Core Problem
核心问题在于如何建立可靠的偏好奖励信号，指导流式视频模型生成符合人类偏好的内容。现有偏好数据多为早期低分辨率视频，难以反映现代高质量视频的细节和动态特征。同时，奖励模型设计缺乏系统性，容易受到偏差和操控，导致偏好对齐效果不稳定。此外，流式模型的特殊机制（如Velocity预测）使得传统对齐方法难以直接应用，亟需创新的对齐策略。
Innovation
本研究的创新点包括：1）构建了182k偏好标注的多维度视频偏好数据集，涵盖视觉质量、运动质量和文本对齐；2）提出VideoReward多维奖励模型，有效捕捉人类偏好；3）设计Flow-DPO、Flow-RWR两种训练策略，避免奖励操控和过拟合；4）引入Flow-NRG推理方法，支持个性化目标调节，提升模型适应性。这些创新突破了流式模型偏好对齐的技术瓶颈，为高质量视频生成提供了理论和实践基础。
Methodology
- �� 构建偏好数据集：采集16k prompts，利用12个T2V模型生成108k视频，标注182k偏好三元组。
- �� 训练奖励模型：采用Qwen2-VL-2B，比较Bradley-Terry（BT）和点估分（Regression）两种偏好学习方式，发现BT优越。
- �� 多维偏好标注：专业标注员对偏好三元组进行视觉质量、运动质量和文本对齐的偏好标注，并赋予Likert评分。
- �� 模型设计：引入多维奖励模型VideoReward，分析设计参数对性能的影响。
- �� 对齐算法：提出Flow-DPO（无时序依赖）、Flow-RWR（奖励加权回归）和Flow-NRG（推理引导），结合KL正则化实现偏好对齐。
- �� 实验验证：在VideoGen-RewardBench和GenAI-Bench上进行评估，比较不同策略的偏好匹配率和视频质量指标。
Experiments
采用多场景数据集，比较不同奖励模型和对齐策略。指标包括偏好准确率、运动平滑性、内容一致性。设置超参数如视频分辨率（448×448）、帧率（2fps）、偏好标注数量（182k）等。进行消融实验验证模型设计的有效性，分析不同KL正则化策略对偏好对齐的影响。还测试了Flow-NRG在个性化目标调节中的表现，确保模型的鲁棒性和泛化能力。
Results
VideoReward在偏好匹配和多维指标上优于现有模型，偏好准确率达72.89%，比基线高出20%以上。Flow-DPO在多维偏好指标中表现最优，避免奖励操控，视频运动更平滑、内容更符合文本描述。Flow-NRG实现了推理阶段的个性化调节，用户可以动态调整多目标权重，满足不同偏好。整体结果验证了多维偏好引导在流式视频生成中的有效性和实用性。
Applications
该技术可广泛应用于虚拟现实、影视后期制作、个性化内容定制等场景，提升生成内容的质量和用户满意度。需要配合高质量偏好数据和强大的计算资源，适合大型内容平台和研究机构采用。未来还可结合自动偏好采集和多目标优化，推动行业创新。
Limitations & Outlook
模型依赖大量偏好标注，数据采集成本高，偏好偏向可能存在偏差。复杂场景和长视频生成仍存在运动不自然、细节不足的问题。推理阶段的调节机制虽灵活，但需手动调参，缺乏自动化调优。未来应提升数据效率，增强模型泛化能力，降低成本。
Plain Language Accessible to non-experts
想象你在一家工厂，每天都在生产不同的商品。以前，工厂只用简单的标准来判断商品好坏，比如外观漂亮或不漂亮，但这些标准不能完全反映顾客的真实需求。后来，工厂开始让顾客试用商品，并告诉工厂哪些方面做得好，哪些需要改进。工厂用这些反馈调整生产流程，使商品更符合顾客的喜好。类似地，这篇论文用人类的偏好数据训练一个“评判员”，让它能更好地判断生成的视频是否符合人们的期待。通过不断学习和调整，生成的视频变得更自然、更流畅，也更符合用户的个性化需求，就像工厂根据顾客反馈不断改进产品一样。
ELI14 Explained like you're 14
想象你在学校里，老师让你评价几份作业，告诉老师哪一份更棒，或者两份都一样。老师收集了很多你的评价，然后用这些评价训练一个“聪明的机器人”，让它也能帮忙判断作业的好坏。这个机器人还能根据你的偏好调整，比如你喜欢画得漂亮还是内容丰富。论文里做的事情类似：他们让很多人评价由不同AI模型生成的视频，然后用这些评价训练一个“奖励模型”，告诉AI模型哪些视频更符合人类的喜好。接着，他们设计了几种方法，让AI模型在生成视频时更好地“听取”这些偏好，从而生成更自然、更吸引人的视频。最后，用户还能在生成时自己调节偏好，让视频更符合自己的口味。这就像你可以在点歌时选择喜欢的歌曲风格一样，模型变得更聪明、更贴心了。
Abstract
Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs.