Graph-GRPO: Training Graph Flow Models with Reinforcement Learning

Graph generation has become a cornerstone in computational chemistry, drug discovery, and materials science, yet existing models face significant challenges in efficiently optimizing complex, goal-specific structures. Traditional flow-based models like DeFoG rely heavily on Monte Carlo sampling, which breaks gradient flow, limiting their integration with reinforcement learning. Consequently, these models struggle to align generated graphs with human preferences or specific objectives, especially in high-dimensional spaces. To address this, the authors introduce Graph-GRPO, a novel framework that derives an explicit analytical expression for the transition probabilities of Graph Flow Models (GFMs). This breakthrough allows the seamless integration of reinforcement learning algorithms, such as Group Relative Policy Optimization (GRPO), enabling end-to-end training. The key innovation lies in replacing non-differentiable sampling with a fully differentiable, analytical transition mechanism, ensuring stable gradient flow and efficient optimization.

Complementing this, the paper proposes a local perturbation-based refinement strategy. By randomly perturbing nodes and edges in promising graphs and regenerating them iteratively, the method enhances exploration within high-potential regions of the graph space. This combination of global policy optimization and local refinement results in a powerful approach capable of generating high-quality, goal-aligned graphs with only 50 denoising steps. Extensive experiments on synthetic datasets, molecular docking, and property optimization tasks demonstrate that Graph-GRPO achieves state-of-the-art performance, with validity scores exceeding 95% in minimal steps, outperforming diffusion models and genetic algorithms.

The broader impact of this work is substantial. It paves the way for more intelligent, goal-oriented graph generative models capable of tackling complex problems in drug discovery, protein engineering, and materials design. By bridging the gap between theoretical rigor and practical efficiency, the framework opens new avenues for scalable, accurate, and adaptable structure generation. Despite its strengths, challenges remain, including computational costs and potential local optima in highly multimodal landscapes. Future research will focus on scaling the approach, integrating multi-scale data, and further enhancing exploration strategies, ultimately aiming to realize fully autonomous, goal-driven design systems in industry and academia.