Spectrally-Guided Diffusion Noise Schedules

Diffusion models are generative models based on a stepwise denoising process, which have recently achieved significant progress in image and video generation. Initially proposed by Sohl-Dickstein et al., and later developed into denoising diffusion probabilistic models (DDPM) by Ho et al., these models form the foundation of the current state-of-the-art latent diffusion models (LDM). LDMs operate in the latent space of visual autoencoders, combining efficient generative capabilities with lower computational costs. However, the generative quality of LDMs is inherently limited by the quality of the autoencoder, and they require multi-stage training, adding complexity and training costs. To overcome these limitations, researchers have explored single-stage pixel diffusion models, improving model architecture and training protocols to narrow the performance gap with LDMs. Despite progress, LDMs still demonstrate better generative quality at lower computational costs, partly due to requiring up to an order of magnitude fewer denoising steps than pixel diffusion. Noise scheduling plays a critical role in diffusion models, typically handcrafted as linear or cosine-like curves increasing with time steps. Recent approaches, such as Simple Diffusion, adapt the schedule across resolutions by shifting the curve. This paper proposes a spectrally-guided per-instance noise scheduling method to further enhance generative quality.

The performance of diffusion models heavily relies on the design of noise schedules, which are traditionally handcrafted and require extensive tuning, especially across different resolutions. This manual scheduling is not only time-consuming but also struggles to adapt to the spectral diversity within datasets, leading to a decline in generative quality. Particularly under low-step conditions, traditional noise schedules may apply too much or too little noise, affecting the generative outcome. Therefore, designing a noise scheduling method that can automatically adapt to the spectral properties of each instance is key to improving the generative quality of diffusion models.

The core innovation of this paper lies in proposing a per-instance noise scheduling method based on the spectral properties of images. First, theoretical bounds on the efficacy of minimum and maximum noise levels are derived, designing tight noise schedules that eliminate redundant steps. Second, during inference, a conditional sampling mechanism is introduced to dynamically adjust the noise schedule according to each instance's spectral properties. This method differs from previous global schedules by adapting to the spectral diversity within datasets, significantly improving generative quality. Additionally, experiments validate the method's effectiveness under low-step conditions, particularly in high-resolution image generation.

The methodology of this paper includes the following key steps:

• �� Spectral Analysis: Perform a discrete Fourier transform (DFT) on each input image to compute its radially-averaged power spectral density (RAPSD), capturing the image's spectral properties.

• �� Noise Schedule Design: Based on the RAPSD, derive theoretical bounds on the efficacy of minimum and maximum noise levels, designing tight noise schedules that eliminate redundant steps.

• �� Conditional Sampling: During inference, use a conditional sampling mechanism to dynamically adjust the noise schedule according to each instance's spectral properties.

• �� Experimental Validation: Conduct experiments on the ImageNet dataset to validate the improvement in generative quality under low-step conditions.

The experimental design includes multi-resolution image generation experiments on the ImageNet dataset. The proposed method is compared with the baseline model SiD2, using the same architecture and training protocol. The experiments use Frechet Inception Distance (FID) as the primary evaluation metric to assess the quality of generated images. To verify the effectiveness of the noise scheduling, ablation studies are conducted to analyze the impact of different noise scheduling strategies on generative quality. Additionally, the generative performance across different resolutions is tested to validate the method's robustness.

Experimental results demonstrate that the proposed method significantly improves generative quality under low-step conditions. On the ImageNet dataset, the method achieves approximately 15% better FID scores than the baseline model SiD2 at 32 steps. Furthermore, the new noise schedules adapt well across different resolutions without the need for hyperparameter adjustments, demonstrating robustness. Ablation studies confirm the effectiveness of spectrally-guided noise schedules in reducing noise steps, especially in high-resolution image generation.

The proposed method can be directly applied to image and video generation tasks, particularly in scenarios requiring high-quality outputs, such as film production and advertising design. Due to its ability to maintain high quality under low-step conditions, it is advantageous in computationally constrained environments. Additionally, the method can improve the training efficiency of generative models, reducing training time and costs.

Despite the method's excellent performance under low-step conditions, it may exhibit slight FID degradation at high step counts. Furthermore, the model still requires tuning for different resolutions, particularly concerning loss bias and guidance intervals. These limitations indicate that while spectrally-guided noise scheduling has advantages, further research is needed to address these issues. Future research directions include applying this method to multi-stage generative models and exploring how to integrate loss bias and guidance intervals with spectral properties.