RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

In recent years, large language models (LLMs) have revolutionized natural language processing, enabling applications from chatbots to complex reasoning systems. Central to their advancement is the reinforcement learning with human feedback (RLHF) paradigm, where reward models guide policy optimization. Traditionally, these reward models have been discriminative, assigning scalar scores to responses based on likelihood or preference probabilities. While effective, this approach faces fundamental limitations: it often fails to fully leverage the generative capabilities of LLMs and suffers from calibration issues, leading to unstable training and suboptimal policy updates.

Recognizing these challenges, recent research has shifted towards generative reward models that express preferences through natural language, capturing nuanced judgments beyond scalar scores. However, integrating these models into reinforcement learning remains problematic. Existing methods typically convert preference signals into scalar rewards via probability estimates, which are prone to collapse and calibration errors, undermining the potential advantages of generative models.

This paper introduces a novel approach called Ranking-based Reward Construction (RRC), designed to harness the full power of generative reward models by directly utilizing response rankings. Instead of relying on probability scores, RRC constructs reward signals from the relative ordering of responses, aligning more naturally with the comparative nature of generative preferences. The method employs two key mechanisms: Self-Competitive Ranking (SCR), which compares responses sampled from the current policy to determine relative quality, and Anchor-Guided Ranking (AGR), which introduces a small set of reference responses to scale the ranking process efficiently.

The core innovation lies in transforming pairwise response preferences into stable, order-preserving reward signals that satisfy margin sensitivity. By doing so, RRC addresses the instability and calibration issues faced by probability-based reward signals, leading to more robust and effective reinforcement learning. Extensive experiments on benchmarks such as AlpacaEval2 and ArenaHardV2 demonstrate that RRC significantly outperforms traditional reward construction methods, achieving improvements of over 5% in success rates and showcasing better scalability when increasing inference time and anchor responses.

The significance of this work extends beyond performance metrics. It fundamentally redefines how preferences expressed by generative models are utilized in reinforcement learning, emphasizing the importance of relative ranking over absolute scoring. This shift unlocks new possibilities for training more aligned, human-like AI systems capable of nuanced understanding and decision-making. The approach also opens avenues for future research into multi-modal preferences, efficient ranking algorithms, and broader applications across AI domains.

Despite its promising results, RRC faces challenges such as increased computational costs and the need for more robust preference inference. Future work will focus on optimizing efficiency, extending to multi-modal data, and further theoretical analysis of ranking properties. Overall, this research marks a significant step toward more natural, stable, and scalable reinforcement learning with large language models, promising a new era of intelligent systems that better understand and align with human preferences.