Flux-OPD: On-Policy Distillation with Evolving Contexts

Recent advances in deep learning have led to powerful models in NLP, vision, and multimodal tasks. Knowledge distillation (KD) has become a key technique to transfer knowledge from large teachers to smaller students, improving efficiency and generalization. Traditional KD methods like vanilla teacher-student training focus on static models and fixed datasets, but struggle in open-ended domains where task preferences and contexts are dynamic. To address this, context-aware distillation methods such as Context Distillation, OPCD, and OEL emerged, allowing models to incorporate privileged information and adapt to evolving environments. However, these approaches face challenges: static contexts become outdated as the student improves, causing training instability and conflicting supervision signals. The need for a framework that can handle evolving contexts, stabilize training, and capture complex preferences motivated this research, leading to the development of Flux-OPD, which combines theoretical insights with practical mechanisms.

The core challenge is how to incorporate evolving, complex contexts into knowledge distillation without destabilizing training. Existing methods either fix contexts before training, leading to outdated supervision, or rely on offline updates, which cause abrupt shifts and conflicts. In open-ended tasks like video generation and medical QA, preferences change over time, and conflicting signals among context-conditioned teachers can lead to oscillations, divergence, or suboptimal convergence. Therefore, designing a stable, adaptive distillation framework that dynamically models context evolution, mitigates conflicts, and maintains training stability remains an open problem. Addressing this is crucial for deploying AI systems capable of continuous learning and personalized adaptation in real-world scenarios.

The paper introduces several key innovations: 1) Reverse KL geometric mean decomposition, which clarifies the student’s target as the geometric mean of context-conditioned teachers and isolates the conflict term; 2) Flux-OPD framework, which iteratively extracts contexts from student trajectories, then injects task preferences via contextual difference signals into a stable teacher anchor; 3) Conflict-aware weighting, which uses the conflict term to modulate correction strength, ensuring stable training amid context shifts. These innovations differ from prior static or offline methods by enabling real-time, adaptive distillation that accounts for context conflicts, thus improving stability and task-specific performance in open-ended domains.

• �� Context extraction: sample trajectories from the current student, extract experience items, and form a context pool. • Context conditioning: compute the geometric mean of teacher distributions conditioned on each context, capturing task preferences. • Contextual correction: interpolate between the stable context-free teacher and the geometric mean, injecting preference signals via a correction coefficient. • Conflict modulation: calculate the conflict indicator from the reverse KL decomposition, then adjust correction strength using a monotonic function, e.g., clipping or scaling. • Training objective: minimize the reverse KL divergence between student and the corrected teacher distribution, iterating across multiple rounds within a single training run. • Implementation: employ multi-turn context updates, adaptive correction, and conflict-aware weighting to ensure stability and adaptability.

Experiments involved two tasks: prompt optimization for video generation and medical QA, using datasets like VPO and RaR-Medicine. Multiple student-teacher pairs (e.g., Qwen3-VL-Instruct 8B and 4B) were tested. Baselines included OPD, OPCD, and OEL. Metrics covered task scores, robustness, and training stability. The models underwent multi-round training with context updates every 300 steps. Ablation studies examined the effects of context evolution, correction, and conflict weighting. Results showed that Flux-OPD consistently outperformed baselines, with +2-3 score improvements, especially under significant context shifts. The method demonstrated robustness across different initializations and configurations.

Flux-OPD achieved superior performance across all tasks, with notable improvements in total scores (e.g., 80.18 vs. 79.28 in VBench). Ablation confirmed that context evolution and conflict-aware weighting significantly enhance stability and accuracy. The multi-round training demonstrated consistent gains, validating the framework’s robustness. The experimental results indicate that dynamically modeling evolving contexts and mitigating conflicts lead to more stable and task-aligned models, especially in complex, preference-rich environments.

This approach is applicable to personalized AI assistants, adaptive content creation, and multi-modal systems where user preferences evolve over time. It enables models to continuously adapt to changing contexts, improving relevance and user satisfaction. Future applications include integrating reinforcement learning for automated context evolution, expanding to larger models, and deploying in real-world scenarios such as healthcare, entertainment, and education, where preferences are dynamic and complex.

The method relies on high-quality context extraction; noisy or biased contexts can impair performance. Handling extreme conflicts remains challenging, potentially causing instability. Computational costs increase with context size and iteration count, limiting scalability. Further research is needed to optimize efficiency, extend to other modalities, and improve conflict detection and resolution mechanisms.