AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward

Recent advancements in Unified Multimodal Models (UMMs) have significantly improved visual understanding and generation. These models, through unified architectures, can seamlessly integrate visual understanding and generation capabilities. However, effectively unlocking these models' intrinsic reasoning capabilities to enhance generation quality remains a challenge. Traditional methods often require an additional cold-start stage, relying on distillation from stronger teacher models, which not only increases computational costs but may also limit model generalization. In recent years, Group Relative Policy Optimization (GRPO) has achieved success in the field of reinforcement learning, particularly in enhancing reasoning capabilities in large language models (LLMs) and optimizing visual generation. The AlphaGRPO framework applies GRPO to AR-Diffusion Unified Multimodal Models, proposing a novel method to enhance multimodal generation capabilities without an additional cold-start stage.

A core problem in multimodal generation models is providing stable supervision signals for generating high-quality visual content. Traditional scalar reward mechanisms often fail to accurately evaluate complex user requests, leading to suboptimal generation results. Moreover, many existing methods rely on distillation from stronger teacher models, increasing computational costs and potentially limiting model generalization. Therefore, effectively activating the model's intrinsic reasoning capabilities to enhance generation quality without increasing computational costs is a pressing issue.

The core innovation of AlphaGRPO lies in the introduction of the novel Decompositional Verifiable Reward (DVReward) mechanism. • DVReward decomposes complex user requests into verifiable semantic and quality questions evaluated by a general MLLM, providing reliable feedback. • This mechanism not only addresses the issue of traditional scalar reward mechanisms failing to accurately evaluate complex requests but also avoids reliance on distillation from stronger teacher models. • Additionally, AlphaGRPO activates the model's reasoning capabilities without an additional cold-start stage, significantly enhancing multimodal generation performance.

The implementation of the AlphaGRPO framework includes several key steps: • First, apply Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models to enhance multimodal generation capabilities. • Then, introduce the Decompositional Verifiable Reward (DVReward), which decomposes complex user requests into verifiable semantic and quality questions evaluated by a general MLLM, providing reliable feedback. • In reasoning text-to-image generation tasks, the model actively infers implicit user intents, and in self-reflective refinement, it autonomously diagnoses and corrects misalignments in generated outputs. • Through experimental validation, AlphaGRPO demonstrates outstanding performance across multiple multimodal generation benchmarks, validating its self-reflective reinforcement approach.

In experimental design, AlphaGRPO is validated across multiple multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench, and WISE. • In experiments, AlphaGRPO achieves 83.9% on TIIF-Bench in reasoning text-to-image tasks, outperforming Bagel by 5.8%. • Furthermore, in editing tasks on GEdit, AlphaGRPO achieves significant gains without training on editing tasks. • Through self-reflective refinement, AlphaGRPO further elevates performance in reasoning text-to-image tasks, validating its self-reflective reinforcement approach.

Experimental results show that AlphaGRPO demonstrates outstanding performance across multiple multimodal generation benchmarks. • On TIIF-Bench, AlphaGRPO achieves 83.9% in reasoning text-to-image tasks, outperforming Bagel by 5.8%. • In editing tasks on GEdit, AlphaGRPO achieves significant gains without training on editing tasks, indicating its broad applicability in multimodal tasks. • Through self-reflective refinement, AlphaGRPO further elevates performance in reasoning text-to-image tasks, validating its self-reflective reinforcement approach.

Application scenarios for AlphaGRPO include but are not limited to: • In multimodal generation tasks, AlphaGRPO can be used to generate high-quality visual content that meets complex user requests, applicable in industries such as advertising design and film production. • In editing tasks, AlphaGRPO can be used to autonomously diagnose and correct misalignments in generated outputs, improving editing efficiency, applicable in image processing software. • In industry, AlphaGRPO provides more efficient solutions for multimodal generation, reducing computational costs, applicable in industries requiring efficient visual content generation.

Despite its outstanding performance in multimodal generation tasks, AlphaGRPO has some limitations. • In complex scenarios, the model may struggle to fully understand implicit user intents, leading to less accurate generation results. • The Decompositional Verifiable Reward mechanism requires multiple MLLM inferences, potentially increasing computational overhead. • In certain specific tasks, performance improvements may be limited, requiring further optimization. Future research directions include further optimizing the Decompositional Verifiable Reward mechanism to reduce computational overhead, exploring applicability in more task scenarios, and integrating other reinforcement learning methods to enhance model generalization.