Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion

Recent advances in diffusion models, such as DDPM, Score-based Models, and transformer-based architectures, have significantly improved image synthesis quality. Early efforts focused on latent diffusion to reduce computational costs, but this introduced external dependencies and potential information loss. Pixel-space diffusion models, especially with large transformers and coarse pixel tokenization, have gained attention for their ability to generate high-fidelity images directly at pixel level. However, these models implicitly learn a coarse-to-fine hierarchy driven by the spectral properties of natural images, where low frequencies contain most of the signal, and high frequencies are often noise. Despite this, the spectral structure remains underutilized, leading to inefficient capacity allocation. Prior work in spectral methods often involved post-processing or spectral regularization in the loss function, but lacked a unified, input-side spectral control aligned with the diffusion process. This paper bridges this gap by explicitly incorporating spectral priors into the input, guided by natural image statistics.

The core challenge in pixel-space diffusion is the implicit coarse-to-fine spectral hierarchy, which causes models to waste capacity on frequency bands dominated by noise, especially at early diffusion stages. The moving spectral boundary k*(t) separates signal-rich low frequencies from noise-dominated high frequencies, but existing models lack explicit mechanisms to leverage this structure. Consequently, models must internally discover this boundary, leading to inefficient training and suboptimal capacity utilization. This results in slower convergence, lower image fidelity, and difficulty in scaling to high-resolution data. Addressing this problem requires explicit spectral guidance to focus model capacity on meaningful signal regions, reducing computational waste and improving overall performance.

The key innovation is the introduction of Spectral Forcing (SF), a parameter-free, time-conditioned 2D-DCT low-pass filter applied at the input. SF explicitly enforces a spectral boundary c(t) that expands monotonically with diffusion time, matching the theoretical boundary k*(t) derived from natural image spectra. Unlike prior methods, SF directly manipulates the input spectrum without learnable parameters, ensuring minimal overhead (~0.5%) and seamless integration. The schedule f(t) controlling c(t) can be linear or analytical, approximating the natural spectral boundary. This approach effectively restricts the model’s view to signal-dominant bands, reducing capacity wastage and accelerating training. The method is validated through synthetic and real-image experiments, demonstrating consistent improvements in image quality and transferability to text-to-image models.

• �� Derive the spectral boundary k*(t) = (1−t)^(-2/α) from natural image power-law spectra P(k) ∝ k^(-α).
• �� Define per-band data-to-noise ratio DNR(k, t) = P(k)/(1−t)^2, establishing a moving cutoff frequency.
• �� Design a parameter-free, time-conditioned 2D-DCT low-pass filter SF_t(z) = IDCT(DCT(z) ⊙ M(t)), where M(t) is a soft mask based on radius r(u,v) and cutoff c(t).
• �� Schedule c(t) to grow monotonically, e.g., linearly or analytically proportional to (1−t)^(-2/α), ensuring the filter matches the spectral boundary.
• �� Apply SF to noisy input z_t before patch embedding, leaving the diffusion process, loss, and sampling unchanged.
• �� Conduct synthetic toy experiments and real ImageNet training to analyze the capacity allocation, spectral boundary effectiveness, and performance gains.
• �� Integrate SF into existing models like SenseNova-U1 to test transferability and downstream improvements.

The experiments used ImageNet-256, training JiT models with 130M and 700M parameters, across multiple epochs (15, 60, 120). Baseline models without SF were compared against SF-enhanced models, evaluating FID and Inception Score. Coarse (64 tokens) and fine (256 tokens) tokenizations were tested to assess the impact of spectral filtering at different granularities. Synthetic toy experiments validated the theoretical spectral boundary k*(t) and capacity allocation, confirming that outside the wedge region models tend to predict deterministically. In real training, SF consistently reduced FID by 8-14% and improved Inception Score by 10-13% across scales and epochs. Additional ablation studies compared different schedule shapes and alternative frequency filtering methods, demonstrating SF’s superior performance and efficiency.

Applying SF on ImageNet-256 with JiT-700M/32 reduced FID from 24.19 to 20.68 (+14.5%) and increased Inception Score from 83.28 to 93.96 (+13%) at 60 epochs. The performance gains persisted across training stages, with the largest improvements in coarse tokenization models. In the integrated SenseNova-U1 model, SF improved downstream metrics like DPG-Bench and GenEval, confirming transferability. Synthetic experiments revealed the capacity wastage outside the signal-recovery wedge, which SF explicitly restricts, leading to more efficient learning. The ablation studies confirmed that time-dependent, data-driven spectral boundary matching yields the best results, outperforming static low-pass filters and other frequency domain methods.

This spectral filtering approach is immediately applicable to high-resolution image generation, especially in scenarios with coarse pixelization or low-frequency dominant signals. It can be integrated into existing pixel-space diffusion pipelines to improve training efficiency and output quality. Long-term, the method can be extended to video, 3D data, and multi-modal generation tasks, enabling more resource-efficient models that better exploit spectral priors. Its transferability suggests potential in industry applications like content creation, virtual reality, and AI-assisted design, where high fidelity and computational efficiency are critical.

The method may suppress essential high-frequency details in images requiring fine textures, reducing fidelity. It relies on the assumption that natural images follow a power-law spectrum, which may not hold in specialized datasets, limiting generality. The additional DCT filtering introduces a slight computational overhead (~0.5%), which could be non-trivial in resource-constrained environments. Future work should explore adaptive spectral boundaries and multi-scale filtering to address these issues.