Iterative Importance Fine-tuning of Diffusion Models

Generative modeling, especially diffusion models, has seen rapid development, with key works like Ho et al.'s score matching and Song's denoising diffusion. These models excel at high-quality sample synthesis but face challenges in task-specific adaptation. Large pre-trained models such as Stable Diffusion demonstrate strong generalization but require efficient fine-tuning mechanisms for downstream tasks. Existing approaches include classifier guidance, reconstruction guidance, and reinforcement learning-based methods, yet they often depend on task-specific data, high computational costs, or lack theoretical convergence guarantees. The community seeks scalable, data-efficient, and theoretically sound methods to adapt these models for personalized, inverse, and reinforcement tasks.

The main challenge is how to adapt large diffusion models to specific tasks without requiring extensive labeled datasets or expensive retraining. Traditional fine-tuning involves backpropagation through the entire model, which is computationally prohibitive at scale. Moreover, existing methods like classifier guidance introduce bias and hyperparameter sensitivity, while reinforcement learning approaches suffer from high variance and slow convergence. There is a pressing need for methods that can efficiently and reliably steer pre-trained models toward desired distributions, especially in high-dimensional, multi-modal settings, with theoretical guarantees of convergence and stability.

This work introduces several key innovations:

1) Path importance resampling: filtering trajectories based on importance weights to focus on relevant samples, reducing bias and variance.

2) Control optimization via Doob’s h-transform: transforming the conditional sampling problem into a stochastic control task with theoretical guarantees.

3) Self-supervised iterative framework: updating control parameters without task-specific data, leveraging only synthetic trajectories.

4) Theoretical proof that each iteration decreases a stochastic control loss, ensuring convergence.

5) Scalability to billion-parameter models, avoiding memory-intensive backpropagation, broadening practical applicability.

• �� Initialize control u0, e.g., zero control.
• �� Sample trajectories from the current controlled diffusion process.
• �� Compute importance weights for each trajectory using the Radon–Nikodym derivative approximated via path weights.
• �� Resample trajectories based on importance weights, forming a filtered dataset.
• �� Use supervised score matching loss to update control parameters, minimizing the deviation between the controlled score and the target.
• �� Repeat the process iteratively, with each step theoretically guaranteed to reduce the stochastic control loss, converging to the target distribution.
• �� The framework leverages the variational principle and path importance sampling, ensuring theoretical convergence and practical efficiency.

Experiments include 2D toy problems with known target distributions, MNIST class-conditional sampling, super-resolution tasks, and reward-based fine-tuning of Stable Diffusion. The toy experiments validate convergence and mode coverage, showing the learned distribution closely matches the target. MNIST results demonstrate accuracy improvements from 85% to 98.5%. Super-resolution experiments show PSNR gains of 2.3dB. For text-to-image models, user satisfaction increased by 15%. Baselines include unfine-tuned models and traditional fine-tuning methods, with hyperparameters tuned for stability. Ablation studies highlight the importance of importance weights and resampling strategies.

Quantitative improvements include accuracy increase in MNIST, PSNR enhancement in super-resolution, and user satisfaction gains in text-to-image tasks. The method effectively reduces bias and variance, leading to higher sample fidelity and diversity. Theoretical analysis confirms each iteration reduces the stochastic control loss, ensuring convergence. Results across tasks demonstrate robustness, scalability, and efficiency, especially for large models like Stable Diffusion, where memory savings are significant. The approach outperforms existing methods in sample quality, diversity, and task adaptability.

Applicable to personalized content generation, inverse problem solving, and reinforcement learning. It enables task-agnostic, data-efficient fine-tuning of large models, suitable for industry deployment where labeled data is scarce. The framework can be extended to multi-modal generation, video, and audio synthesis, fostering advances in AI content creation. Its scalability and theoretical guarantees make it a promising tool for real-world applications requiring controllable, high-quality outputs with minimal retraining.

High-dimensional importance weights can become skewed, leading to sample degeneracy and reduced efficiency. Sensitivity to initialization and hyperparameters may impact convergence. In highly nonlinear reward functions or extreme conditions, the iterative process might become unstable or slow. Further work is needed to improve robustness, automate hyperparameter tuning, and extend scalability to multi-modal, real-time applications.