Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
RVM fine-tunes diffusion velocity fields without trajectories, matching or exceeding policy-gradient baselines at lower cost; no numerical scores are provided.
Key Findings
Methodology
The paper proposes Reward-based Velocity Matching (RVM), which updates a diffusion model's velocity field directly. It avoids stochastic denoising trajectory likelihoods and endpoint ELBO approximations. High-reward generations reinforce their associated directions, low-reward generations suppress them, and an optional reference-velocity anchor controls drift. RVM also subsumes RAM and DiffusionNFT as special cases.
Key Results
- Across several large-scale diffusion reward-fine-tuning tasks, RVM is reported to match or outperform trajectory-based policy-gradient methods while using substantially less training cost. The supplied text gives no exact scores, percentage gains, or compute figures.
- For video generation, standard preference rewards can favor visually clean but nearly static outputs. The proposed dynamic-tracking reward improves motion while also improving overall VBench performance, although no numerical VBench values are included.
- Ablations indicate that after simplifying the velocity update, the choice among loss variants matters less than reward construction and anchor design, identifying objective specification as the dominant practical factor.
Significance
RVM reframes diffusion reward fine-tuning as optimization in the model's native velocity representation rather than likelihood optimization. This bypasses the long-standing difficulty that diffusion samples lack tractable endpoint likelihoods. By reducing trajectory construction and probability-ratio machinery, the method offers a simpler route to scalable preference adaptation for image and video generators. It also questions whether policy-gradient techniques inherited from autoregressive language models are necessary for diffusion systems.
Technical Contribution
The work contributes a trajectory-free velocity update, reward-weighted reinforcement and suppression of velocity directions, and an optional reference-velocity anchor for controlling distribution drift. It gives a unifying view in which RAM and DiffusionNFT arise as special cases. In addition, it treats reward design—not merely optimization loss—as a central technical component for video, introducing dynamic tracking to counteract static-video failure modes.
Novelty
The fundamental novelty is a change in optimization object: RVM operates on the velocity field instead of reconstructing the probability of a sampled denoising trajectory. Relative to RAM, DiffusionNFT, and related likelihood-based policy-gradient approaches, it is direct, trajectory-free, and designed for scaling. The supplied material does not establish a formal first-in-field claim or provide new theoretical guarantees.
Limitations
- The provided abstract and text omit datasets, model sizes, training steps, hardware, exact VBench scores, and statistical tests. Therefore the magnitude and reproducibility of the reported cost and quality advantages cannot be independently assessed.
- RVM remains dependent on reward quality. A reward that overvalues cleanliness, aesthetics, or static appearance can induce systematic exploitation; an anchor limits drift but cannot correct a semantically misaligned reward.
Future Work
Future work should systematically study reward families, anchor strengths, velocity parameterizations, and multi-objective conflicts while reporting compute and uncertainty. Important directions include stability theory for reward-weighted velocity updates, adaptive anchoring, long-video temporal consistency, online human feedback, reward hacking detection, and integration with few-step diffusion samplers.
AI Executive Summary
Diffusion models now generate highly realistic images and videos, but aligning them with human preferences remains technically awkward. Existing methods borrow policy-gradient machinery from language models: they either assemble stochastic denoising transitions into a trajectory likelihood or approximate endpoint likelihood with an evidence lower bound. Because diffusion samples do not expose tractable likelihoods, these strategies add computation and algorithmic complexity.
Choi and colleagues propose Reward-based Velocity Matching (RVM), which updates the diffusion velocity field directly. Directions associated with high-reward samples are reinforced, low-reward directions are suppressed, and an optional reference-velocity anchor limits departure from the pretrained model. The framework recovers RAM and DiffusionNFT as special cases, presenting a common formulation rather than another isolated objective.
Across large-scale diffusion tasks, the authors report that RVM is competitive with or better than trajectory-based policy-gradient methods at substantially lower cost, although the supplied text contains no numerical table. In video, ordinary preference rewards can produce clean but nearly motionless clips; a dynamic-tracking reward improves motion and overall VBench performance. The broader message is that reward fine-tuning may be better posed in velocity space, while reward and anchor design matter more than elaborate loss variants.
Deep Analysis
Background
Diffusion generation can be viewed as a sequence or continuous-time evolution from noise to data. Recent methods such as RAM and DiffusionNFT adapt these models with rewards, but they commonly import likelihood-ratio and policy-gradient ideas. Autoregressive models expose token probabilities; diffusion models do not provide equally tractable sample likelihoods, forcing trajectory-based transition products or ELBO approximations.
Core Problem
The central problem is how to shift a diffusion model toward high-reward outputs without reconstructing complete denoising trajectories or paying for expensive likelihood estimation. The method must also control distribution drift and handle reward misspecification. Video adds a further conflict: rewards for visual cleanliness may ignore meaningful temporal motion.
Innovation
RVM changes the optimization target from trajectory probability to velocity. It reinforces velocity directions linked to high rewards, suppresses low-reward directions, and optionally anchors the update to a reference velocity. This unifies RAM and DiffusionNFT and extends the design space through dynamic-tracking rewards for video, rather than relying solely on generic visual preference scores.
Methodology
- �� Input: a reference diffusion model, generated samples, time states, and scalar rewards.
- �� Velocity matching: predict the time-conditioned velocity and adjust its learning signal according to sample reward.
- �� Reward shaping: emphasize directions from high-reward generations and de-emphasize those from low-reward generations.
- �� Anchoring: penalize deviation from the reference velocity to balance adaptation and preservation.
- �� Unified view: particular weighting and anchoring choices recover RAM and DiffusionNFT.
- �� Video extension: add a dynamic-tracking reward to explicitly value temporal motion.
Experiments
The study evaluates RVM on several large-scale diffusion reward-fine-tuning tasks and compares it with trajectory-based policy-gradient approaches. Video experiments use VBench for overall assessment and contrast standard preference rewards with dynamic tracking. Ablations vary velocity-update losses, reward design, and anchoring. The supplied text does not specify datasets, backbone sizes, hyperparameters, sampling steps, or compute budgets.
Results
RVM is reported to achieve competitive or superior quality with substantially reduced training cost. Once the velocity update is simplified, loss variants contribute less than reward and anchor choices. In video, dynamic tracking reduces the tendency toward clean but static outputs and improves motion alongside overall VBench performance. Exact numerical comparisons are unavailable in the provided material.
Applications
The method applies to text-to-image, image editing, and text-to-video preference adaptation, including aesthetics, instruction following, motion quality, and task-specific rewards. Deployment requires a reliable reward model, a reference model, and an evaluation protocol. Its trajectory-free structure is especially attractive for large-scale experimentation and iterative product tuning.
Limitations & Outlook
The supplied text lacks complete experimental tables, so robustness across model families, reward scales, and data regimes cannot be judged. RVM can still optimize reward-model biases, while anchoring introduces a trade-off: excessive anchoring limits improvement, whereas weak anchoring may damage quality, diversity, or alignment. Future work should add theory, multi-objective control, longer-video evaluation, and transparent compute accounting.
Plain Language Accessible to non-experts
Imagine a painter who turns a blurry canvas into a finished picture through many small decisions. Traditional training records every decision from start to finish and then tries to calculate how likely the entire sequence was to produce a good painting. That is slow and difficult. RVM takes a simpler view: at each moment, it asks which direction the brush moved. If the final picture received praise, that direction becomes more likely next time; if it received criticism, the direction is weakened.
The optional anchor is the painter's original style guide. It prevents the artist from chasing a score so aggressively that every picture becomes strange. Video exposes another issue: a judge may reward a sharp, beautiful scene even if nothing moves. Dynamic tracking is like adding a judge who explicitly applauds natural action, so the system learns that a good film should be both clear and alive.
ELI14 Explained like you're 14
Picture an AI teammate in a game that turns a fuzzy image into a finished drawing one step at a time. Older training methods save a replay of every move and calculate how likely the whole replay was. That works more naturally for chatbots than for image generators, so it becomes expensive and messy.
RVM skips the giant replay review. It looks at the direction the AI took at each step. If the final picture gets a high score, the AI is encouraged to take similar directions; if it gets a low score, those directions are discouraged. An optional reference model acts like an older, reliable teammate saying, “Improve, but do not become totally weird!”
Videos reveal a funny trap: a judge may love a clean picture, so the AI makes a beautiful car that barely moves. The dynamic-tracking reward tells it that movement matters too. The paper says this improves motion and overall VBench results.
The big idea is surprisingly simple: adjust the AI’s next-step directions instead of calculating the probability of an entire noisy journey. But there is a catch: the scoreboard still matters. If the reward is bad, the AI may become extremely good at chasing the wrong goal!
Glossary
Reward-based Velocity Matching (RVM)
A method that adjusts a diffusion model's velocity field using rewards from generated samples. It avoids explicitly constructing full denoising trajectory likelihoods.
The paper's central algorithm.
Velocity field
A function specifying how a diffusion state should evolve at each time. It is a native representation of continuous diffusion generation.
The object directly optimized by RVM.
Trajectory-free
An update that does not require storing and evaluating a complete stochastic denoising path. This can reduce sampling and likelihood-estimation overhead.
A main distinction from policy-gradient baselines.
Anchor term
A regularizer encouraging the new velocity field to remain close to a reference velocity. It controls drift from the pretrained model.
An optional RVM component.
RAM
A recent diffusion reward-fine-tuning method discussed by the paper. Under the proposed formulation, it can be recovered as a special case.
A related baseline and unifying example.
DiffusionNFT
Another diffusion reward-fine-tuning method included in the paper's conceptual comparison. It is also represented as a special case of RVM.
A related method and baseline.
VBench
A benchmark suite for video-generation quality, covering visual and temporal dimensions. It provides an aggregate evaluation framework rather than a single perceptual judgment.
Used for overall video assessment.
Open Questions Unanswered questions from this research
- 1 The supplied text omits exact scores, compute savings, and significance tests, leaving the magnitude of RVM's advantage over each baseline unresolved.
- 2 A general theory connecting reward scale, anchor strength, velocity parameterization, and training stability is still missing.
- 3 It remains unclear how dynamic rewards generalize to long videos, multi-agent interactions, and diverse human preferences.
Applications
Immediate Applications
Video motion alignment
Video-model developers can add a dynamic-tracking reward to existing preference rewards and apply RVM to the velocity field. A reference model, evaluable reward, and metrics such as VBench are required; the expected benefit is less bias toward attractive but nearly static clips.
Low-cost image preference tuning
Image-generation services can use aesthetic, instruction-following, safety, or task rewards with RVM without constructing full denoising trajectories. This supports rapid product iteration while preserving a reference model as a quality safeguard.
Long-term Vision
A unified preference-optimization layer for generators
RVM could become a common interface for image, video, and 3D generation by mapping diverse rewards directly into velocity updates. Realizing this vision will require standard reward protocols, multi-objective controls, stability theory, and robust defenses against reward hacking.
Abstract
Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing methods largely inherit policy-gradient machinery from large language models. Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples. As a result, current approaches either construct trajectory likelihoods from stochastic denoising transitions or approximate endpoint likelihoods with evidence lower bound, introducing additional computation and algorithmic complexity. We demonstrate that this likelihood-based machinery is not necessary for effective diffusion reward fine-tuning. We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field. RVM reinforces directions associated with high-reward generations, suppresses those with low reward, and involves an optional anchor term controlling drift from a reference velocity. Notably, it provides a general framework that recovers recent fine-tuning methods, including RAM and DiffusionNFT, as special cases. Across various large-scale diffusion models reward fine-tuning tasks, RVM is competitive with or outperforms trajectory-based policy-gradient methods under substantially reduced training cost. We further find that, once the velocity update is simplified, the particular loss variant matters less than reward and anchor design. For video generation, standard preference rewards can favor visually clean but nearly static outputs; introducing a new dynamic-tracking reward that substantially improve motions while improving overall VBench performance. These results suggest that scalable reward fine-tuning for diffusion models is better posed in the native velocity representation than as likelihood-based policy optimization.