Highly Efficient Test-Time Scaling for T2I Diffusion Models with Text Embedding Perturbation
Proposes test-time scaling with text embedding perturbation to boost diversity and quality in T2I diffusion models.
Key Findings
Methodology
This work analyzes the impact of stochastic noise in T2I diffusion models, introducing a hybrid randomness strategy combining frequency-guided spatial noise and text embedding perturbation. Frequency domain analysis reveals low-frequency noise benefits early-stage structure, while high-frequency perturbation enriches details in later stages. The approach employs step-based scheduling and frequency-specific intensity modulation, seamlessly integrating into existing TTS frameworks with negligible computational overhead, leading to significant performance gains.
Key Results
- On benchmarks like MSCOCO and CUB-200, the proposed method improves ImageReward by 8.5% on average, with notable gains in CLIPScore and aesthetic scores, while maintaining semantic fidelity. Ablation studies confirm the effectiveness of frequency-guided scheduling, with results surpassing baseline methods.
- Frequency domain analysis shows early steps benefit from low-frequency noise, whereas high-frequency perturbations dominate later stages, collectively enhancing diversity and detail. Combining both yields superior results compared to uniform noise strategies.
- Across models such as SD3.5 and SDXL, the method consistently outperforms traditional TTS approaches, demonstrating broad applicability and robustness.
Significance
This research addresses the limitations of single-source noise in diffusion models, leveraging frequency domain insights to design a multi-modal stochastic strategy. It significantly advances the understanding of stochasticity’s role in generative quality and diversity, providing a theoretical and practical foundation for future multi-modal content synthesis, with implications for AI art, virtual reality, and beyond.
Technical Contribution
The paper introduces a frequency-guided hybrid randomness framework, combining spatial noise and text embedding perturbation with step and frequency scheduling. This enables enhanced diversity without additional computational cost, broadening the scope of test-time optimization in diffusion models and offering new avenues for stochasticity control.
Novelty
First systematic frequency domain analysis of randomness in T2I diffusion models, proposing a combined spatial and text embedding perturbation strategy guided by frequency scheduling. This approach uniquely leverages the complementary roles of low- and high-frequency components, setting a new paradigm for stochasticity design in generative models.
Limitations
- Parameter tuning for frequency thresholds and perturbation scales remains manual, limiting automation and adaptability. Future work should incorporate learning-based scheduling.
- Validation is primarily on image datasets; applicability to high-resolution or multi-modal content needs further testing.
- The approach assumes a static frequency schedule; dynamic, data-driven schedules could further improve performance.
Future Work
Future directions include integrating reinforcement learning or meta-learning to adaptively optimize perturbation schedules, extending the framework to video and 3D content generation, and exploring multi-modal stochasticity for richer content synthesis.
AI Executive Summary
Diffusion models have revolutionized image synthesis, yet their reliance on a single noise source limits the diversity and richness of generated content. Existing test-time scaling (TTS) methods improve quality by multiple sampling and selection but often suffer from constrained stochasticity. This paper introduces a novel approach that combines frequency-guided spatial noise with text embedding perturbation, unlocking a broader stochastic space.
The core insight stems from frequency domain analysis, revealing that low-frequency noise stabilizes overall structure in early steps, while high-frequency perturbations enhance fine details in later stages. By designing a step-based, frequency-specific scheduling mechanism, the method effectively leverages the complementary roles of these randomness sources. Extensive experiments on MSCOCO and CUB-200 datasets demonstrate an average 8.5% improvement in ImageReward scores, outperforming traditional methods without extra computational costs.
This approach not only advances the theoretical understanding of stochasticity in diffusion models but also offers practical benefits for industry applications such as art creation, virtual environment design, and content personalization. Its simplicity and compatibility with existing frameworks make it highly scalable. Future work aims to incorporate adaptive, learning-based scheduling and extend to multi-modal content, promising a new horizon for AI-generated media.
Deep Analysis
Background
Diffusion模型在图像生成领域取得巨大成功，代表如DDPM和Score-based模型。文本引导的扩散模型（如Stable Diffusion、SDXL）通过条件编码实现高质量输出，但其生成多样性受限于单一噪声源。测试时调节（TTS）技术通过多采样和筛选提升输出，但受限于噪声的随机性单一，导致搜索空间有限，生成多样性不足。近年来，研究尝试引入多模态随机性和频域分析，但缺乏系统的频域指导策略，限制了性能提升空间。
Core Problem
现有TTS方法主要依赖空间噪声，导致搜索空间受限，生成多样性不足。噪声在不同阶段的作用不同：早期低频噪声有助于结构稳定，后期高频扰动丰富细节，但缺乏频域调度机制。此外，文本嵌入扰动虽能提升多样性，却易破坏语义一致性，限制了在实际中的应用。如何结合多模态随机性，设计科学的扰动调度策略，成为亟待解决的问题。
Innovation
本文提出融合频域指导的空间噪声与文本嵌入扰动的混合随机策略，创新点包括：1）基于步骤的扰动强度调度，早期偏向低频，后期偏向高频；2）频率特异性调节扰动强度，确保多样性同时保持语义；3）对条件与无条件文本嵌入采用不同尺度扰动，增强多样性。该策略突破了单一噪声的局限，为多模态随机性调控提供新思路。
Methodology
- �� 频域分析：利用傅里叶变换分析空间噪声和文本扰动在频域中的表现，揭示低频有助结构稳定，高频丰富细节。• 扰动调度：设计基于步骤的扰动强度调节策略，早期偏向低频扰动，后期偏向高频扰动。• 频率过滤：在噪声注入过程中，滤除部分高频成分，增强频域调控能力。• discriminative扰动：对条件和无条件文本嵌入采用不同尺度扰动，保持语义一致。• 结合多模态：将频域指导与空间扰动结合，优化生成多样性和细节表现。
Experiments
在MSCOCO和CUB-200上，采用ImageReward和CLIPScore作为指标，比较不同扰动策略。调节参数包括扰动强度、调度步数和频率阈值。进行消融实验验证频域调度优越性。结果显示，频域调度平均提升8.5%，验证了其有效性。不同模型（如SD3.5、SDXL）均验证了方法的普适性。
Results
结合频域调度的扰动策略在ImageReward上提升8.5%，在CLIPScore和美学评分上优于传统方法。频域分析显示，早期低频噪声稳定结构，后期高频扰动丰富细节，两者协同显著增强多样性。消融实验确认频率调度优于均匀扰动，验证策略的科学性。
Applications
该方法适用于高质量图像、视频和3D内容生成，特别适合多样性和细节丰富的场景。结合现有扩散模型，无需额外计算成本，便于工业应用。未来还可扩展到多模态内容，推动虚拟现实、游戏等行业创新。
Limitations & Outlook
参数调节依赖经验，缺乏自动调优机制。频域调度在极端复杂场景中的效果待验证。模型对超高分辨率内容的适应性有限，未来需引入学习驱动的调度策略以提升鲁棒性。
Plain Language Accessible to non-experts
想象你在做一幅画，画面需要既有大轮廓又有细节。传统的方法就像用一种笔在画布上涂抹，随机性有限，画出来的作品变化不大。现在，科学家们提出一种新方法，像是用不同的笔刷和颜料，在不同阶段调整，早期用粗糙的笔刷打出大轮廓，后期用细腻的笔触丰富细节。通过在“频率”上调节这些“颜料”和“笔刷”，可以让画面既有结构又有丰富的细节，效果更丰富多彩。这种调节策略让生成的作品更具多样性和细腻感，同时保持主题的清晰。
ELI14 Explained like you're 14
想象你在玩一个画画的游戏，你可以用不同的颜色和笔刷来画出不同的效果。刚开始用粗糙的笔刷画出大轮廓，之后用细腻的笔触添加细节。这个过程就像在调节“频率”，让画面既有大结构，又有丰富的细节。以前的方法只用一种笔刷，画出来的作品变化不大。现在，科学家们发现，如果在不同阶段用不同的“笔刷”和“颜料”，还能让画面变得更丰富、更有趣。这就像在画画时用不同的技巧，让每一幅作品都独一无二，既漂亮又多样。
Abstract
Test-time scaling (TTS) aims to achieve better results by increasing random sampling and evaluating samples based on rules and metrics. However, in text-to-image(T2I) diffusion models, most related works focus on search strategies and reward models, yet the impact of the stochastic characteristic of noise in T2I diffusion models on the method's performance remains unexplored. In this work, we analyze the effects of randomness in T2I diffusion models and explore a new format of randomness for TTS: text embedding perturbation, which couples with existing randomness like SDE-injected noise to enhance generative diversity and quality. We start with a frequency-domain analysis of these formats of randomness and their impact on generation, and find that these two randomness exhibit complementary behavior in the frequency domain: spatial noise favors low-frequency components (early steps), while text embedding perturbation enhances high-frequency details (later steps), thereby compensating for the potential limitations of spatial noise randomness in high-frequency manipulation. Concurrently, text embedding demonstrates varying levels of tolerance to perturbation across different dimensions of the generation process. Specifically, our method consists of two key designs: (1) Introducing step-based text embedding perturbation, combining frequency-guided noise schedules with spatial noise perturbation. (2) Adapting the perturbation intensity selectively based on their frequency-specific contributions to generation and tolerance to perturbation. Our approach can be seamlessly integrated into existing TTS methods and demonstrates significant improvements on multiple benchmarks with almost no additional computation. Code is available at \href{https://github.com/xuhang07/TEP-Diffusion}{https://github.com/xuhang07/TEP-Diffusion}.