Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training
Proposes GRPO with JS divergence to balance performance and diversity in LlamaGen and Janus-7B.
Key Findings
Methodology
This study extends GRPO's divergence from fixed KL to the f-divergence family, including forward KL, reverse KL, and JS divergence. JS divergence emerges as the best trade-off mechanism.
Key Results
- JS divergence achieves top metrics like HPS-v2.1 (0.2881) on Janus-7B, outperforming other divergences.
- JS maintains high diversity, e.g., 1D entropy of 7.5081 on Janus-7B.
- JS effectively balances performance and diversity, avoiding excessive policy deviation or mode collapse.
Significance
This work addresses the challenge of balancing performance and diversity in text-to-image generation, offering a novel optimization perspective for GRPO frameworks.
Technical Contribution
Introduces a unified f-divergence framework, analyzes gradient impacts, and systematically compares JS divergence with other divergences in GRPO.
Novelty
First to integrate JS divergence into GRPO, providing theoretical and empirical evidence for its advantages in balancing performance and diversity.
Limitations
- Relies on high-quality reference policies; poor references may degrade performance.
- Experiments are limited to LlamaGen and Janus-7B, lacking broader scalability tests.
- Generalization to other tasks remains unexplored.
Future Work
Future work could explore JS divergence in larger models, multimodal tasks, and dynamic divergence parameter adjustments.
AI Executive Summary
Text-to-image (T2I) generation has advanced rapidly, but aligning outputs with human preferences remains challenging. Existing GRPO methods often use fixed KL divergence for regularization, overlooking alternative divergences.
This paper introduces a unified f-divergence framework for GRPO, systematically analyzing forward KL, reverse KL, and JS divergence. Theoretical and experimental results demonstrate that JS divergence achieves the best balance between performance and diversity. Experiments on LlamaGen and Janus-7B show that JS divergence outperforms others in metrics like CLIPScore and HPS-v2.1 while maintaining high diversity.
This study offers a new perspective on optimizing T2I generation, particularly in selecting divergence forms for GRPO. Future work could extend these findings to larger models and multimodal tasks, unlocking further potential in generative AI.
Deep Analysis
Background
Text-to-image generation has gained traction, with autoregressive models emerging as an alternative to diffusion models. These models predict discrete visual tokens but face challenges in aligning outputs with human preferences.
Core Problem
Existing GRPO methods fix KL divergence for regularization, ignoring its impact on optimization dynamics. This can lead to outputs that deviate from human preferences or lack diversity.
Innovation
This paper introduces f-divergence to GRPO, analyzing forward KL, reverse KL, and JS divergence. JS divergence uniquely balances performance and diversity by modulating gradient updates.
Methodology
- �� Proposes a unified f-divergence framework.
- �� Analyzes gradient impacts of different divergences.
- �� Validates on LlamaGen and Janus-7B, comparing metrics like CLIPScore and HPS-v2.1.
Experiments
Experiments use HPS-v2.1 as the reward model, evaluating metrics like CLIPScore and diversity measures. Models include LlamaGen and Janus-7B.
Results
JS divergence achieves top performance on metrics like HPS-v2.1 (0.2881) and maintains high diversity, e.g., 1D entropy of 7.5081 on Janus-7B.
Applications
Applicable to T2I tasks in advertising and content creation, optimizing alignment with human preferences while preserving diversity.
Limitations & Outlook
Relies on high-quality reference policies; scalability and generalization to other tasks need further exploration.
Plain Language Accessible to non-experts
Imagine a chef following a recipe. The recipe is like the reference policy, and divergence determines how much the chef can deviate. JS divergence acts like a flexible guide, allowing creativity while ensuring quality and variety.
ELI14 Explained like you're 14
Think of it like a game where AI helps you draw. Old methods are like strict teachers who make you follow rules. This new method is like a cool coach, letting you be creative while still making awesome art!
Glossary
GRPO (Group-wise Relative Preference Optimization)
A reinforcement learning method using group sampling and relative rewards to optimize generative models.
Used for aligning text-to-image generation with human preferences.
JS Divergence (Jensen-Shannon Divergence)
A metric for measuring similarity between two distributions, balancing between forward and reverse KL.
Used to balance performance and diversity in this paper.
CLIPScore
A metric for evaluating how well generated images match text descriptions.
Key performance metric in experiments.
HPS-v2.1
A reward model for measuring human preference alignment in generated outputs.
Used as the reward signal in GRPO.
Autoregressive Generation
A method of generating discrete tokens step-by-step, with each step depending on the previous one.
The main generation framework studied in this paper.
Open Questions Unanswered questions from this research
- 1 How can divergence parameters be dynamically adjusted during training?
- 2 How does JS divergence perform in larger-scale models?
- 3 Can this approach generalize to other tasks like text generation?
Applications
Immediate Applications
Advertising Design
Optimizes image generation for diverse and high-quality advertisements tailored to user needs.
Content Creation
Provides creators with tools for generating high-quality, diverse images.
Long-term Vision
Multimodal Generation
Extends the method to tasks like joint text and audio generation, enabling richer content creation.
Abstract
Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging. GRPO-style online reinforcement learning provides an effective framework; however, existing methods typically treat reference-policy divergence as fixed, despite its direct impact on policy optimization. We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence, for GRPO-style autoregressive T2I alignment. Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways. In particular, under the sampled-token shaping form used, JS regularization achieves a favorable trade-off by mitigating uniform bias relative to the reference policy while still discouraging large deviations. Extensive experiments on LlamaGen and Janus-7B show that JS divergence achieves the strongest or highly competitive optimization performance on most evaluation metrics while maintaining favorable generation diversity. The code is available at https://github.com/tuoyou-hao/BPD-GRPO.