When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
Introducing perceptual entropy to prevent diversity collapse in flow-based RLHF, achieving a score of 0.734 and diversity of 0.989, surpassing baselines.
Key Findings
Methodology
This paper analyzes why policy entropy remains constant in flow-based RLHF despite diversity collapse, attributing it to fixed noise schedules and mode-seeking optimization. It proposes perceptual entropy, measured in perceptual space via VAE and encoders like DINO and CLIP, as a new regularization metric. Two strategies—Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space (PCVAE)—are designed to enhance diversity. The approach is validated on datasets FLUX.dev and SD3.5-M, with metrics including Vendi Score, reward, and perceptual variance. Results show significant improvements in both quality and diversity, with PEC achieving an overall score of 0.734 and diversity of 0.989, outperforming baseline methods.
Key Results
- Perceptual entropy exhibits an exponential relationship with reward (R=−a exp(Hperc)+b), serving as an effective diversity indicator. Incorporating perceptual entropy regularization boosts overall scores from 0.366 to 0.734 and diversity from 0.047 to 0.989 across multiple models and perceptual spaces.
- Experiments demonstrate that the proposed strategies maintain high-quality generation while preventing mode collapse. The models explore multiple high-reward regions, avoiding the narrow focus typical of mode-seeking algorithms.
- Compared to traditional policy entropy regularization, perceptual entropy aligns better with perceptual space variations, leading to more balanced exploration and content diversity. The methods are robust across neural and rule-based reward settings.
Significance
This work addresses a fundamental challenge in flow-based RLHF—policy entropy's inability to reflect true perceptual diversity—by proposing perceptual entropy. It bridges a gap between theoretical understanding and practical needs for diverse content generation. The approach enhances the applicability of flow models in real-world scenarios like art, design, and content creation, where diversity is crucial. The integration of perceptual space metrics offers a new paradigm for regularizing generative models, potentially influencing future research in multimodal AI and reinforcement learning.
Technical Contribution
The paper introduces perceptual entropy, a novel metric capturing perceptual diversity in a learned feature space, overcoming policy entropy's limitations. It establishes a theoretical relationship between perceptual entropy and reward, similar to findings in LLMs. The design of PEC and PCVAE strategies demonstrates how to incorporate perceptual diversity directly into RL objectives, ensuring exploration across multiple high-reward modes. The work combines information theory, perceptual encoding, and reinforcement learning to develop a comprehensive framework for diversity preservation.
Novelty
This is the first work to incorporate perceptual entropy into flow-based RLHF, directly addressing the policy entropy's invariance issue. Unlike prior methods relying solely on policy entropy regularization, this approach measures diversity in perceptual space, enabling models to explore multiple high-reward regions. The integration of VAE and perceptual encoders like DINO and CLIP for diversity measurement is a novel contribution, providing a new theoretical and practical framework for multimodal content generation.
Limitations
- The effectiveness of perceptual entropy depends on the quality of pre-trained perceptual encoders, which may not generalize well across different modalities or tasks. In some cases, the perceptual space may not fully capture all aspects of diversity.
- Hyperparameter tuning (λ) for regularization strength is sensitive; improper settings can lead to instability or limited diversity gains. Computational costs increase with multiple perceptual encoders and large models.
- The current framework is primarily validated on image generation tasks; extending to other modalities like audio or video requires further adaptation and validation. Future work should explore more efficient perceptual metrics and broader applications.
Future Work
Future research will focus on developing adaptive perceptual encoders that better capture task-specific diversity, integrating multi-modal perceptual spaces, and optimizing computational efficiency. Exploring reinforcement learning strategies that dynamically balance reward maximization and diversity, especially in high-dimensional spaces, is also promising. Extending the framework to video and audio generation, as well as real-time interactive systems, could significantly broaden its impact.
AI Executive Summary
Flow-based models have revolutionized text-to-image generation, yet their fine-tuning through Reinforcement Learning from Human Feedback (RLHF) often leads to a collapse in content diversity. Traditionally, policy entropy has been used as a proxy for diversity, encouraging exploration and variety in generated samples. However, this paper uncovers a critical flaw: in flow models, policy entropy remains fixed during training due to the rigid noise schedule, even as perceptual diversity diminishes. This paradox hampers the effectiveness of entropy regularization, as models tend to mode-seek, converging on narrow high-reward regions and losing the richness of variation.
To address this, the authors propose a novel concept—perceptual entropy—that measures diversity directly in a perceptual feature space derived from VAE and encoders like DINO and CLIP. This metric faithfully reflects perceptual diversity and remains sensitive to mode collapse. Building on this insight, two strategies are introduced: Perceptual Entropy Constraint (PEC), which maximizes perceptual entropy to promote exploration, and Perceptual Constraints on Generation Space (PCVAE), which aligns the perceptual and generative spaces. These methods are integrated into the RLHF framework, resulting in significant improvements.
Experimental results across datasets FLUX.dev and SD3.5-M demonstrate that the new approach boosts overall scores from 0.366 to 0.734 and enhances diversity from 0.047 to 0.989. The models effectively explore multiple high-reward regions, avoiding mode collapse and producing more varied, high-quality images. The findings establish perceptual entropy as a powerful tool for balancing quality and diversity in multimodal generative models.
Despite these advances, challenges remain. The reliance on pre-trained perceptual encoders introduces dependency on their robustness and generalization. Hyperparameter tuning and computational costs are non-trivial, especially for large-scale applications. Future work aims to refine perceptual metrics, extend to other modalities, and develop adaptive strategies for dynamic diversity management, promising a broader impact on AI content creation and multimodal learning.
Deep Dive
Abstract
RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning. In RL, diversity is often assumed to correlate with policy entropy, motivating entropy regularization. However, we show this intuition breaks in flow models: policy entropy remains constant, even while perceptual diversity collapses. We explain this mismatch both theoretically and empirically: the constant entropy arises from the fixed, pre-defined noise schedule, while the diversity collapse is driven by the mode-seeking nature of policy gradients. As a result, policy entropy fails to prevent the model from converging to a narrow high-reward region in the perceptual space. To this end, we introduce perceptual entropy that captures diversity in a perceptual space and maintains the property of standard entropy. Building upon this insight, we propose two entropy-regularized strategies, Perceptual Entropy Constraint and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve the quality. Experiments across two base models, neural and rule-based rewards, and three perceptual spaces demonstrate consistent gains in the quality-diversity trade-off; PEC achieves the best overall score of 0.734 (vs. baseline's 0.366); a complementary setting of PEC further reaches a diversity average of 0.989 (vs. baseline's 0.047). Our project page (https://xiaofeng-tan.github.io/projects/PEC) is publicly available.