FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models
FlowCTS replaces KL-OPD with trajectory supervision, raising GenEval to 0.93, OCR to 0.92, and PickScore to 23.06.
Key Findings
Methodology
FlowCTS initializes student and reference trajectories from the same student-visited state, then matches their subsequent ODE evolution. Using the integral relation between states and velocity fields, it derives a temporally weighted velocity-matching upper bound and discretizes it with K Euler supervision steps. K=1 gives single-state velocity matching; K>1 supervises multiple future states. The same formulation supports on-policy OPD and off-policy SFT.
Key Results
- With SD3.5-Medium and task-specific Flow-GRPO references, FlowCTS-OPD(K=1) improves over Vanilla OPD from 0.90 to 0.93 on GenEval, 0.90 to 0.92 on OCR, and 22.75 to 23.06 on PickScore; Overall rises from 0.890 to 0.912.
- K=2 reaches GenEval 0.94, OCR 0.92, and PickScore 22.96, while outperforming FlowGRPO-Mix at 0.73/0.83/21.84. It also converges faster than KL-based OPD.
- Off-policy FlowCTS-SFT improves OCR from Vanilla SFT's 0.70 to 0.75 at K=3. FlowCTS-OPD also raises ImageReward from 1.21 to 1.27 and HPS-v2.1 from 0.30 to 0.31.
Significance
The work transfers the central OPD idea—dense supervision on the learner's own state distribution—from autoregressive language models to continuous flow generators. It shows that post-training quality depends not only on the reference model or reward, but also on where supervision is allocated along denoising time. For industry, reference-guided trajectory matching offers a potentially cheaper alternative to sparse-reward online optimization while preserving balanced multi-objective capabilities and improving convergence.
Technical Contribution
The paper derives a trajectory-based upper bound, LCTS≤LVUB, with continuous weight w(r)=[(ts−te)^2−(ts−r)^2]/2 and discrete weight αi=[K(K+1)−i(i+1)]/2. These weights arise from error accumulation rather than manual tuning. The formulation unifies single-state velocity matching, multi-step trajectory supervision, on-policy OPD, and off-policy SFT, while exposing supervision horizon K as a new scaling dimension.
Novelty
Unlike Flow-OPD, which reformulates deterministic ODE dynamics as auxiliary SDE transition kernels and minimizes their KL divergence, FlowCTS directly compares continuous future trajectories initialized at a shared student state. Its fundamental innovation is replacing an artificial local stochastic transition target with accumulated velocity-field evolution and an explicit trajectory horizon K.
Limitations
- Increasing K is not consistently beneficial: online K=3 reduces PickScore to 22.60, and offline K=4 falls below K=3. Longer rollouts amplify early prediction errors, increase optimization difficulty, and may suppress fine details.
- Evidence is concentrated on SD3.5-Medium, Flow-GRPO references, image generation, and 50K Fine-T2I examples. Robustness to larger models, alternative flow parameterizations, and other modalities remains untested.
Future Work
Promising directions include adaptive K and timestep weighting based on uncertainty, prompt complexity, or student–reference discrepancy; more efficient shared integration; and stronger analysis of ODE/SDE rollout choices. The method should also be tested on video, 3D, multimodal conditioning, larger generators, and reference ensembles with varying quality.
AI Executive Summary
On-policy distillation has become an effective response to sparse rewards and exposure bias in language-model post-training. Yet flow-based image generators pose a different problem: they evolve through continuous velocity fields rather than autoregressive next-token distributions. Existing Flow-OPD methods therefore construct auxiliary stochastic differential-equation transition kernels and optimize KL divergence. The paper argues that this can misallocate supervision toward the final image, even though the largest student–reference differences emerge during early denoising, when global structure is formed.
FlowCTS—Flow Continuous Trajectory Supervision—starts from a state actually visited by the student and lets both student and reference velocity fields continue along their respective reverse-time trajectories. It minimizes the distance between these future trajectories. An integral argument converts this objective into a tractable, temporally weighted velocity-matching upper bound. A K-step Euler discretization makes the horizon explicit: K=1 is direct velocity matching at one state, while larger K supervises subsequent student and reference states and uses weights αi=[K(K+1)−i(i+1)]/2.
Experiments with SD3.5-Medium and three Flow-GRPO experts show clear gains. Relative to Vanilla OPD, K=1 improves GenEval from 0.90 to 0.93, OCR from 0.90 to 0.92, and PickScore from 22.75 to 23.06; K=2 reaches GenEval 0.94. FlowCTS also exceeds FlowGRPO-Mix and improves broader quality metrics and OCR in offline Fine-T2I supervision. However, longer horizons are not automatically better: K=3 lowers online PickScore to 22.60. The study therefore frames temporal supervision as a central design choice for flow-model post-training.
Deep Analysis
Background
Flow matching trains a time-dependent velocity field to transport noise into data through an ODE. Modern systems such as SD3.5-Medium can be post-trained with reward optimization, including DDPO, DPO-style preference methods, Dance-GRPO, and Flow-GRPO. Language-model OPD adds dense teacher supervision on student-generated prefixes, but flow models lack a natural next-step conditional distribution. Flow-OPD addresses this through auxiliary SDE kernels, leaving open whether the resulting temporal weighting matches the true locations of student–reference disagreement.
Core Problem
The precise problem is to supervise a flow model on states induced by its own rollout while making its future evolution agree with a reference model. This is difficult because states are continuous, velocity fields depend on time and state, errors accumulate through integration, and the student and reference follow different paths after the shared starting state. An SDE-derived KL may emphasize late refinement despite early structural errors being more consequential.
Innovation
- �� CTS formulates OPD as distance between future student and reference trajectories from one shared state.
- �� The Velocity Upper Bound converts trajectory discrepancy into weighted velocity MSE.
- �� K-step Euler supervision unifies one-state and multi-step matching.
- �� The starting-state distribution is modular: student rollouts yield FlowCTS-OPD, while offline data yields FlowCTS-SFT.
- �� Temporal analysis identifies a mismatch in KL-Flow-OPD: its auxiliary-SDE weight concentrates effective loss near the image endpoint.
Methodology
- �� Sampling: in OPD, draw xts and condition c from the current student's SDE rollout; in SFT, construct them from offline Fine-T2I trajectories.
- �� Continuous dynamics: define xθt=Φθts→t(xts,c) and xreft=Φrefts→t(xts,c).
- �� Trajectory target: minimize LCTS=E∫tets||xθt−xreft||²dt.
- �� Upper bound: integrate velocity differences to obtain LVUB=E∫tetsw(r)||vθ−vref||²dr, where w(r) increases toward the segment start because earlier errors affect more future states.
- �� Discretization: with h=(ts−te)/K, update xi+1=xi−hv(xi,ti,c), then minimize Σαi||vθ−vref||² with normalized αi.
- �� Routing: prompt subsets for GenEval, OCR, and PickScore are sent to their corresponding Flow-GRPO references under shared rollout and optimization settings.
Experiments
The on-policy study uses SD3.5-Medium, task-specific Flow-GRPO checkpoints, student SDE rollouts, and comparisons with pretrained SD3.5-Medium, specialized experts, FlowGRPO-Mix, and Vanilla OPD with KL. Primary metrics are GenEval, OCR, PickScore, and Overall; additional tests use T2I-CompBench, ImageReward, Aesthetic Score, UnifiedReward, and HPS-v2.1. The off-policy study uses 50K Fine-T2I image–text pairs, LoRA, Vanilla SFT, and FlowCTS-SFT with K=2,3,4. K, rollout type, and broader generalization are analyzed.
Results
FlowCTS-OPD K=1 reaches 0.93/0.92/23.06 on GenEval/OCR/PickScore versus Vanilla OPD's 0.90/0.90/22.75 and FlowGRPO-Mix's 0.73/0.83/21.84. K=2 gives 0.94/0.92/22.96 and Overall 0.914. On T2I-CompBench it improves Shape to 0.6285, 3D-Spatial to 0.4433, and Numeracy to 0.6909. K=3 offline raises OCR to 0.75 from 0.70, but K=4 declines, confirming a non-monotonic horizon trade-off.
Applications
FlowCTS is directly useful for consolidating multiple image-generation experts into one model, such as jointly improving text rendering, composition, and aesthetic preference. Teams with existing Flow-GRPO checkpoints can use them as dense references without optimizing sparse rewards at every step. Organizations with high-quality image–text trajectories can apply FlowCTS-SFT to improve spatial structure and OCR. The main prerequisites are accessible reference velocity fields, compatible flow parameterization, and sufficient compute for paired rollouts.
Limitations & Outlook
The method requires reference-model evaluations and additional student–reference integration, making it more expensive than single-state SFT. Larger K supplies richer future information but compounds rollout errors and may suppress perceptual details: online PickScore falls to 22.60 at K=3, while offline K=4 underperforms K=3. Experiments use one main base model, image-generation benchmarks, selected Flow-GRPO references, and 50K offline pairs, so cross-architecture and cross-modality generality is unresolved. Future work should learn adaptive horizons and reduce integration overhead.
Plain Language Accessible to non-experts
Imagine a cooking school with a student chef and an expert chef. Both are handed the same half-prepared dish. A conventional teacher may judge only the final plate, or compare one artificial mini-step. If the student made a mistake while arranging the ingredients at the beginning, the final inspection comes too late: every later action has already been affected.
FlowCTS asks both chefs to continue from the same half-prepared dish and compares what they do over the next few actions. Early actions receive more attention because an early mistake influences many later actions. K=1 is like checking only the next move; K=2 or K=3 means watching a short sequence of moves.
The experiments show that this makes the image generator learn faster and improves text, shape, and composition. But watching too far ahead is not always useful. Once the student makes an early mistake, later comparisons become noisy and harder to follow. Thus the best teaching plan balances richer information against confusion. The central lesson is simple: for a step-by-step creator, the timing and path of feedback can matter as much as the final score.
ELI14 Explained like you're 14
Think of an image generator as a game character walking through a maze. The character starts in a noisy, confusing place and gradually builds a picture. A teacher character knows a better route. Ordinary training might only check the final treasure room, or compare one made-up move. If you turn left too early, the later mistakes pile up.
FlowCTS puts the student and teacher at the same checkpoint, then watches them move toward the goal. It compares their directions at the next few checkpoints, not just the final picture. Early directions matter more because they shape everything that comes afterward. K=1 means checking one move; K=2 or K=3 means checking a short chain.
On SD3.5-Medium, ordinary OPD scored 0.90 on GenEval, while FlowCTS K=1 reached 0.93. OCR rose from 0.90 to 0.92, and PickScore from 22.75 to 23.06—pretty impressive! K=2 even reached 0.94 on GenEval.
But longer is not automatically better. At K=3, PickScore dropped to 22.60 because early mistakes spread through later checkpoints. It is like a coach giving so many instructions that the player loses focus. The cool idea is that the model learns the route to a good image, not only how to imitate the final screenshot.
Glossary
Flow Matching
A way to generate data by continuously transporting noise through a learned velocity field. In this paper, the field follows an ODE and is trained with a velocity target such as x1−x0.
It provides the generative foundation for SD3.5-Medium and FlowCTS.
On-policy Distillation
Distillation performed on states generated by the current student rather than states sampled only from the teacher. It supplies dense reference feedback where the student actually operates.
FlowCTS-OPD samples its shared starting states from student rollouts.
Continuous Trajectory Supervision
A training objective that compares the future continuous paths of student and reference models initialized from the same state. It replaces a natural next-token or next-step distribution with path-level supervision.
CTS is the paper's central formulation.
Velocity Upper Bound
A tractable weighted velocity-error objective that upper-bounds the distance between student and reference trajectories. Its weights reflect how long each velocity error affects later states.
It converts the integral trajectory loss into model-output regression.
Flow-GRPO
An online policy-optimization method for flow models using stochastic sampling trajectories and reward feedback.
Its task-specific checkpoints provide references, and FlowGRPO-Mix is an RL baseline.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether K and timestep weights can be selected automatically from prompt difficulty, uncertainty, or student–reference disagreement; the current study only establishes that fixed horizons behave non-monotonically.
- 2 The evidence is limited to image generation. Video, 3D, multimodal conditioning, larger models, and the cost of using many reference models require systematic evaluation.
Applications
Immediate Applications
Multi-objective text-to-image distillation
Image platforms can route prompts for GenEval, OCR, and PickScore to specialized reference checkpoints while training one student with FlowCTS. This can balance capabilities without directly optimizing several sparse rewards, provided reference velocity fields and rollout compute are available.
Trajectory-aware offline fine-tuning
Teams with Fine-T2I-like image–text data can replace single-state SFT with FlowCTS-SFT. K=2 or K=3 can provide structural supervision that improves text rendering, spatial relations, and compositional consistency while retaining the original data pipeline.
Long-term Vision
Adaptive flow-model post-training
A future system could choose K and temporal weights from uncertainty, prompt complexity, and predicted structural risk. Such adaptive supervision may reduce paired-integration costs and become a general alignment layer for image, video, and 3D flow generators.
Abstract
While on-policy distillation (OPD) effectively addresses sparse rewards and exposure bias in large language model post-training, its extension to flow models remains underexplored. To this end, we propose Flow Continuous Trajectory Supervision (FlowCTS), which matches subsequent student and reference trajectories initialized from the same student-visited state. Using the integral relation between trajectories and velocity fields, we derive a temporally weighted velocity-matching upper bound and discretize it into practical objectives parameterized by the number of supervision steps. Under a multi-reference setup, single-state FlowCTS-OPD outperforms vanilla KL-based OPD with faster convergence. FlowCTS-OPD improves GenEval from 0.90 to 0.93, OCR from 0.90 to 0.92, and PickScore from 22.75 to 23.06, while outperforming a mixed-reward RL baseline across all target metrics. Further analysis reveals a clear temporal supervision mismatch in vanilla KL-based OPD arising from its auxiliary SDE transition kernels. Beyond on-policy setting,FlowCTS also consistently outperforms vanilla SFT , particularly on OCR, while increasing supervision steps exhibit a trade-off between richer trajectory information and greater optimization difficulty.