GRADE: Personalized Multi-Task Fusion via Group-relative Reinforcement Learning with Adaptive Dirichlet Exploration
GRADE framework uses group-relative policy optimization and Dirichlet exploration for personalized multi-task fusion, improving CTR by 0.595% and CVR by 1.193%.
Key Findings
Methodology
This paper introduces GRADE, a critic-free reinforcement learning framework based on Group Relative Policy Optimization (GRPO). It employs candidate weight groups to evaluate relative performance, avoiding critic instability. Dirichlet distribution is used for exploration in the continuous weight space, ensuring weights are non-negative and sum to one. The approach combines sparse user feedback, dense model priors, and rule-based rewards to guide learning. Training involves supervised pretraining followed by reinforcement fine-tuning, enabling stable and personalized weight adjustment.
Key Results
- In large-scale AB tests, GRADE achieved +0.595% CTR, +1.193% CVR, +1.788% OPM, and +1.568% total order volume, outperforming baselines.
- Deployment in Kuaishou's marketplace demonstrated sustained improvements over static weights, validating industrial applicability.
- Ablation studies confirmed Dirichlet exploration and composite rewards are critical for performance gains, enhancing exploration efficiency and stability.
Significance
This work advances personalized recommendation by replacing static fusion weights with a dynamic, reinforcement learning-based approach. It addresses key industry challenges—training instability and exploration inefficiency—by introducing a robust, scalable framework. The successful large-scale deployment exemplifies its practical impact, offering a new paradigm for adaptive multi-objective optimization in recommender systems, ultimately improving user satisfaction and business metrics.
Technical Contribution
The core innovation lies in adopting a critic-free, group-relative policy optimization that simplifies training and enhances stability. The use of Dirichlet distribution for exploration respects the constraints of fusion weights, and the multi-source reward design enriches learning signals. This combination provides a theoretically sound, practically efficient solution for high-dimensional, continuous parameter optimization in large-scale systems.
Novelty
This is the first application of group-relative policy optimization with Dirichlet exploration in recommendation system fusion tasks. It uniquely combines relative performance evaluation with constrained continuous exploration, overcoming the limitations of traditional Actor-Critic RL methods. The approach introduces a new way to efficiently and stably optimize personalized weights in large-scale industrial environments.
Limitations
- The model heavily depends on abundant user feedback, which may limit effectiveness in cold-start scenarios or for new users.
- Dirichlet sampling can be computationally intensive in very high-dimensional spaces, potentially affecting scalability.
- Current validation is limited to specific recommendation scenarios; generalization to other multi-task settings needs further exploration.
Future Work
Future research will focus on integrating multi-modal data, accelerating exploration via adaptive sampling, and applying meta-learning to enhance rapid adaptation. Expanding to diverse domains and reducing computational costs will further facilitate industrial deployment and broader applicability.
AI Executive Summary
In the rapidly evolving landscape of online recommender systems, balancing multiple objectives such as click-through rate, conversion rate, and order volume remains a significant challenge. Traditional static weighting schemes, while simple, lack the flexibility to adapt to individual user preferences and session dynamics. Reinforcement learning offers a promising avenue for personalized, adaptive weight adjustment, but existing methods—particularly Actor-Critic architectures—suffer from instability and inefficient exploration in high-dimensional continuous spaces.
This paper introduces GRADE, a novel framework that leverages group-relative policy optimization (GRPO) combined with Dirichlet distribution-based exploration to address these issues. By evaluating candidate weight groups relative to each other, GRADE circumvents the instability associated with critic networks, providing a more stable training process. The Dirichlet distribution ensures exploration respects the non-negativity and sum-to-one constraints of fusion weights, enabling efficient search in high-dimensional spaces. The approach employs a multi-source reward function, integrating sparse user feedback, dense model priors, and rule-based heuristics, to guide learning effectively.
Deployed in a large-scale industrial setting with hundreds of millions of daily users, GRADE demonstrated significant improvements over static and prior RL-based methods. In rigorous AB tests, it achieved +0.595% CTR, +1.193% CVR, +1.788% OPM, and +1.568% total order volume, surpassing baseline strategies. Its success in Kuaishou’s marketplace validates its practicality and robustness, offering a scalable solution for personalized multi-objective optimization.
This work marks a substantial step forward in recommendation system research, providing a stable, efficient, and theoretically grounded method for dynamic weight personalization. Future directions include multi-modal data integration, faster exploration algorithms, and broader domain adaptation, aiming to further enhance the system’s responsiveness and generalization capabilities. Despite its achievements, challenges such as cold-start performance and computational costs remain, guiding ongoing research efforts toward more universally applicable solutions.
Deep Dive
Abstract
Balancing multiple objectives is critical for user satisfaction in modern recommender and search systems, yet current Multi-Task Fusion (MTF) methods rely on static, manually-tuned weights that fail to capture individual user intent. While Reinforcement Learning (RL) offers a path to personalization, traditional approaches often falter due to training instability and the sparse rewards inherent in these large-scale systems. To address these limitations, we propose Group-relative Reinforcement learning with Adaptive Dirichlet Exploration (GRADE), a novel and robust framework for personalized multi-task fusion. GRADE leverages a critic-free, Group Relative Policy Optimization (GRPO) paradigm, enabling stable and efficient policy learning by evaluating the relative performance of candidate weight groups. Its core innovations include employing the Dirichlet distribution for principled and structured exploration of the weight space, and a composite reward function that combines sparse user feedback with dense model priors and rule-based constraints to guide the search effectively. Deployed in the in-app marketplace of an application with over hundreds of millions daily active users, GRADE significantly outperforms established baselines, achieving substantial gains in rigorous large-scale A/B tests: +0.595\% in CTR, +1.193\% in CVR, +1.788\% in OPM, and +1.568\% in total order volume. Following its strong performance, GRADE has been fully deployed in the marketplace search scenario of Kuaishou, serving hundreds of millions of users.