Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
D3PO method directly fine-tunes diffusion models using human preferences without reward models, achieving performance comparable to reward-based methods.
Key Findings
Methodology
This work models the denoising process of diffusion models as a multi-step Markov Decision Process (MDP), extending the theoretical framework of Direct Preference Optimization (DPO). The approach bypasses reward model training by directly leveraging human preferences to optimize the policy via Q-value approximation, integrating preference signals into each denoising step. The method employs KL divergence constraints to maintain stability, enabling efficient, low-cost fine-tuning. The core innovation is translating human preferences into policy updates without explicit reward functions, supported by rigorous theoretical proofs of equivalence to reward model training.
Key Results
- Experiments show D3PO achieves comparable improvements in image quality, reduced distortion rates, and enhanced safety, matching or surpassing reward-model-based fine-tuning. Using relative objective scales as preferences, the method reduces hand and full-body deformities by approximately 30%, and improves aesthetic scores by 15%. The training process involves 400 epochs, generating 80 images per epoch, with consistent performance gains across multiple metrics.
- In experiments without reward models, D3PO effectively reduces hand deformities from 50% to 20%, demonstrating strong bias correction. It also improves image safety and alignment with prompts, verified through manual preference annotations. Multi-objective optimization balances compressibility, incompressibility, and aesthetic quality, with high data efficiency.
- Across various tasks, D3PO maintains stable training and outperforms baseline methods, confirming the effectiveness of preference-based policy updates in complex generative tasks. The approach significantly reduces hardware requirements, making large-scale diffusion model fine-tuning feasible.
Significance
This research addresses the high cost and complexity of reward model training in diffusion model fine-tuning, offering a scalable, resource-efficient alternative. By theoretically linking preference optimization to reward learning, it broadens the scope of preference-based reinforcement learning. The method’s ability to improve image realism, safety, and alignment without extensive data or hyperparameter tuning has profound implications for industrial applications such as content creation, safety assurance, and personalized AI. It paves the way for democratizing high-quality generative AI by reducing hardware barriers and simplifying workflows, potentially transforming how large models are adapted for specific tasks.
Technical Contribution
The key technical contribution is the formal extension of DPO into a multi-step MDP framework, enabling direct policy optimization from human preferences without reward models. The authors derive a new loss function based on Q-value approximations, prove the theoretical equivalence to reward model training, and incorporate KL divergence constraints for stability. The algorithm reduces memory consumption by utilizing sub-segment sampling, allowing large diffusion models to be fine-tuned efficiently. These innovations collectively advance the theoretical understanding and practical implementation of preference-driven model optimization.
Novelty
This is the first work to eliminate reward models in diffusion model fine-tuning by directly incorporating human preferences into a policy optimization framework. Unlike prior RLHF or ReFL approaches, which depend on explicit reward models trained on large datasets, D3PO leverages preference signals via Q-value approximation, offering a more scalable and hardware-friendly solution. Its theoretical foundation guarantees optimality and stability, marking a significant step forward in preference-based deep learning.
Limitations
- The method relies heavily on the quality and consistency of human preferences; biased or noisy annotations can impair training effectiveness.
- In highly complex or ambiguous tasks, preference signals may be insufficient or unreliable, limiting model improvements.
- Scaling to extremely large models still faces hardware challenges, especially in preference sampling and data collection, requiring further optimization.
Future Work
Future directions include integrating multi-modal preferences, automating preference collection via active learning, and extending the framework to other generative modalities like video and text. Improving robustness against noisy preferences and exploring semi-supervised approaches could further enhance scalability. Additionally, developing standardized benchmarks for preference-based diffusion model fine-tuning will facilitate broader adoption and community validation.
AI Executive Summary
The rapid evolution of deep generative models has revolutionized content creation, yet fine-tuning these models remains resource-intensive. Traditional approaches like RLHF depend on training complex reward models, which demand vast datasets and extensive hyperparameter tuning, limiting scalability. This bottleneck has hindered widespread deployment of high-quality diffusion models in practical applications. Addressing this challenge, the authors introduce D3PO (Direct Preference for Denoising Diffusion Policy Optimization), a novel framework that leverages human preferences directly to optimize diffusion models without the need for reward models. By conceptualizing the denoising process as a multi-step Markov Decision Process (MDP), D3PO extends the theoretical foundation of preference optimization, enabling end-to-end policy updates based solely on preference signals. The core idea hinges on approximating the human preference distribution through Q-values, which guide the model's parameters to generate higher-quality images aligned with human tastes. The authors rigorously prove that this approach is equivalent to traditional reward-based methods, ensuring theoretical soundness. Extensive experiments on image quality, safety, and deformity reduction demonstrate that D3PO achieves performance comparable to, or better than, reward-model-dependent techniques, while significantly reducing hardware and data requirements. Notably, in experiments without any reward model, the method effectively reduces hand and full-body deformities, improves aesthetic scores, and enhances safety, validating its practical utility. The approach's efficiency and robustness open new avenues for scalable, preference-driven model fine-tuning, with promising applications in digital art, content moderation, and personalized AI. Future work will focus on multi-modal preferences, active learning integration, and broader application domains, aiming to democratize high-quality content generation with minimal resource overhead. Overall, D3PO represents a significant leap toward accessible, efficient, and human-aligned generative AI systems.
Deep Analysis
Background
Deep generative models如GAN、变分自编码器和扩散模型在图像、视频和文本生成中取得巨大突破。扩散模型如DDPM和Stable Diffusion凭借其稳定性和高质量输出成为研究焦点。传统微调方法多依赖奖励模型或大量人类标注，成本高昂且难以扩展。RLHF在大模型中已显示优越性，但在扩散模型中应用受限于高内存和训练复杂性。近年来，偏好优化逐渐兴起，试图绕过奖励模型，直接利用偏好信号进行模型调整，推动偏好学习的研究发展。
Core Problem
扩散模型微调面临多重挑战：一是奖励模型训练成本高，二是模型去噪过程涉及大量中间状态，硬件资源消耗巨大，三是偏好信号稀疏且偏差可能影响训练效果。如何在保证偏好信息充分利用的同时，降低硬件门槛，成为关键难题。传统方法难以满足大规模应用需求，亟需新颖、低成本的微调策略。
Innovation
本文的创新点包括：1）将扩散模型的去噪过程形式化为多步MDP，为偏好优化提供理论基础；2）提出无需奖励模型的偏好直接优化(D3PO)，利用Q值近似偏好信息，实现端到端微调；3）结合KL散度限制偏离参考策略，确保训练稳定性。这些创新突破了依赖奖励模型的限制，显著降低硬件成本，增强模型的适应性和可扩展性。
Methodology
- �� 将扩散模型的去噪步骤定义为多步MDP，状态包括条件信息、时间步和潜在图像；
- �� 利用偏好数据，将偏好转化为策略优化目标，避免训练奖励模型；
- �� 通过引入Q值近似，估算偏好对应的价值函数，指导去噪策略调整；
- �� 在每一步去噪中，利用偏好信息更新模型参数，结合KL散度限制偏离参考策略，确保训练稳定；
- �� 采样偏好由人工标注，逐步优化模型生成能力。
Experiments
采用LAION数据集和Stable Diffusion v1.5模型，验证偏好优化效果。偏好采样由人工标注，训练400轮，每轮生成80张图像。评估指标包括图像质量、畸变率和安全性，比较D3PO与传统奖励模型微调方法。偏好尺度作为目标，验证模型在减少畸变、提升审美方面的能力。无奖励模型的微调在手部畸变和内容安全上表现优越，验证偏好导向的有效性。
Results
D3PO在畸变率降低30%，安全性提升明显，偏好指标与奖励模型方法持平甚至优越。偏好尺度作为代理，达到了与真实奖励相似的优化效果。无奖励微调显著改善手部畸变，从50%降至20%，验证偏好导向的有效性。多目标优化中，模型在压缩性、不可压缩性和审美指标上表现均衡，训练样本利用率高，训练过程稳定。
Applications
该方法适用于内容生成、内容安全、个性化定制等场景，尤其在硬件资源有限或需要快速迭代的应用中表现优越。未来可扩展到视频、文本等多模态内容，推动偏好驱动的无奖励微调体系，促进工业界的广泛应用。
Limitations & Outlook
偏好采集依赖人工标注，偏差可能影响训练效果。偏好信号在复杂或偏好不明确任务中稀疏，模型泛化能力有待提升。大规模模型偏好采样仍面临硬件瓶颈，需优化偏好采样策略和偏好表示。未来应结合主动学习、多模态偏好增强模型鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房里做菜，你想做出最受家人喜欢的菜。每次做完后，你请家人尝一尝，告诉你喜欢哪一幅。你不用学所有菜谱，只根据家人的偏好不断调整调料和火候，逐步做出大家都喜欢的菜。这就像让模型自己学会哪些图片更漂亮，不用事先告诉它评分标准。它通过观察偏好，逐步改进，最后做出符合人们心意的图片。这种方法不用训练复杂的评分器，只用偏好信息，让模型变得更棒，就像厨师不断调整菜肴一样。
ELI14 Explained like you're 14
想象你在玩一个画画游戏，你画了两幅画，想知道哪一幅更酷。你请朋友帮忙选择，他们会告诉你喜欢哪一幅。然后，你根据朋友的偏好调整画风，下次画得更符合他们的喜好。你不用学复杂的评分规则，只听朋友的喜欢与不喜欢，逐步改进。这就像D3PO，不用专门训练评分器，而是直接用人们的偏好让模型变得更漂亮。这样既省事，又能画出大家都喜欢的图片！
Glossary
扩散模型 (Diffusion Model)
一种通过逐步添加和去除噪声生成高质量图像的深度模型，核心机制是逆向扩散过程。
论文中的基础模型，用偏好优化进行微调。
偏好优化 (Preference Optimization)
利用人类偏好信息直接指导模型参数调整的方法，无需训练奖励模型，基于偏好转化为策略优化。
本文提出的核心技术，用于替代RLHF中的奖励模型。
Q值函数 (Q-Function)
在强化学习中，用于估算在某状态采取某行动的预期回报的函数，帮助策略优化。
在D3PO中用来近似偏好对应的价值，指导去噪步骤。
马尔可夫决策过程 (MDP)
描述序列决策问题的数学框架，包括状态、动作、转移概率和奖励。
将扩散模型的去噪过程形式化为多步MDP，作为偏好优化基础。
Open Questions Unanswered questions from this research
- 1 如何在偏好信号稀疏或偏差较大的情况下，保证模型学习的稳定性和鲁棒性仍需深入研究。
- 2 偏好采集的自动化和多模态融合技术尚未成熟，影响偏好信息的丰富性和准确性。
Abstract
Using reinforcement learning with human feedback (RLHF) has shown significant promise in fine-tuning diffusion models. Previous methods start by training a reward model that aligns with human preferences, then leverage RL techniques to fine-tune the underlying models. However, crafting an efficient reward model demands extensive datasets, optimal architecture, and manual hyperparameter tuning, making the process both time and cost-intensive. The direct preference optimization (DPO) method, effective in fine-tuning large language models, eliminates the necessity for a reward model. However, the extensive GPU memory requirement of the diffusion model's denoising process hinders the direct application of the DPO method. To address this issue, we introduce the Direct Preference for Denoising Diffusion Policy Optimization (D3PO) method to directly fine-tune diffusion models. The theoretical analysis demonstrates that although D3PO omits training a reward model, it effectively functions as the optimal reward model trained using human feedback data to guide the learning process. This approach requires no training of a reward model, proving to be more direct, cost-effective, and minimizing computational overhead. In experiments, our method uses the relative scale of objectives as a proxy for human preference, delivering comparable results to methods using ground-truth rewards. Moreover, D3PO demonstrates the ability to reduce image distortion rates and generate safer images, overcoming challenges lacking robust reward models. Our code is publicly available at https://github.com/yk7333/D3PO.