RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
RLAR uses LLMs to autonomously retrieve and synthesize reward models, improving multi-task RL performance by 10-60%.
Key Findings
Methodology
RLAR introduces an agent-driven framework where LLM agents autonomously search internet repositories for high-quality reward models, then synthesize domain-specific verifiers via code generation. The system employs a reward toolset that expands through agentic synthesis, selecting appropriate models via a policy router based on task metadata. It leverages the Group Relative Policy Optimization (GRPO) algorithm to optimize policy updates efficiently, reducing reliance on static reward models. Experiments across math, coding, translation, and dialogue tasks demonstrate performance gains of 10-60%, with RLAR approaching the performance ceiling and outperforming static baselines.
Key Results
- On REWARDBENCH-V2, RLAR boosts average scores by 10.4% on Llama-3.1-8B and 61.9% on Qwen-8B, surpassing all static reward baselines. In GSM8K, it improves accuracy by 13.1 points, and in LeetCode Pass@1, by 6.66 points. The reward tool selection accuracy reaches 90.44%, outperforming SOTA. Cost reductions include 80% API token savings and 75% GPU hours. Ablation studies confirm the importance of reward synthesis and routing modules, with dynamic scheduling ensuring robustness.
- The framework's ability to self-evolve reward tools enables it to adapt to diverse tasks and data shifts, maintaining stable performance across benchmarks. Its reward toolset's accuracy and adaptability contribute to significant improvements in out-of-distribution generalization, reducing reward hacking and format biases. The experimental results validate RLAR’s scalability, efficiency, and superior task performance.
Significance
This work addresses the core challenge of static reward models in multi-task RL, offering a flexible, self-updating reward system that enhances generalization and efficiency. By automating reward model retrieval and synthesis, RLAR reduces reliance on costly annotation and domain expertise, paving the way for more adaptable AI systems. Its ability to self-evolve reward signals aligns with the goal of creating more robust, scalable, and versatile language models capable of handling diverse real-world tasks, thus significantly advancing the field of AI alignment and reinforcement learning.
Technical Contribution
RLAR's key innovation lies in integrating internet-based reward model retrieval with code-based verifier synthesis, enabling reward systems to dynamically adapt during training. The use of GRPO reduces the dependency on value models, improving training stability and efficiency. The reward toolset's self-expansion through agentic synthesis introduces a new paradigm for reward modeling, combining retrieval, synthesis, and validation in a unified framework. These contributions collectively push the boundary of multi-task RL, offering a scalable, cost-effective, and generalizable approach.
Novelty
This is the first framework to combine internet-driven reward model retrieval with automated code synthesis for reward verification, enabling reward systems to self-evolve during training. Unlike prior static or handcrafted reward models, RLAR dynamically constructs task-specific reward tools, significantly improving generalization across diverse domains. Its integration of retrieval, synthesis, and validation mechanisms represents a novel approach that addresses longstanding limitations in reward modeling for large-scale language models.
Limitations
- RLAR's effectiveness depends on the quality of retrieved reward models; if retrieval yields biased or inaccurate models, performance may suffer. Further robustness mechanisms are needed.
- Code verification primarily targets mathematical and programming tasks; extending to perception or multimodal tasks remains challenging.
- In highly dynamic or unseen domains, reward tool synthesis may face difficulties, requiring more adaptive validation methods.
Future Work
Future directions include integrating multimodal reward tools, such as visual or speech-based verifiers, to support broader tasks. Enhancing the robustness of reward synthesis and validation, possibly via meta-learning, is also planned. Additionally, exploring federated or distributed reward evolution could enable scalable, privacy-preserving multi-task RL in real-world applications.
AI Executive Summary
RLAR (Reinforcement Learning from Agentic Rewards) introduces a novel framework where large language models (LLMs) autonomously retrieve and synthesize reward models from the internet, enabling dynamic reward system evolution during training. Traditional reward models rely on static, domain-specific classifiers or judges, which are costly to train and often fail to generalize beyond their training distribution. RLAR addresses this by leveraging LLM agents that perform web searches to find high-quality reward models tailored to specific tasks, such as math reasoning, coding, translation, and dialogue. These models are then synthesized into verifiers via code generation, ensuring high-fidelity reward signals even under data distribution shifts.
The core innovation lies in the reward system's self-evolving capability. Using a policy router, RLAR dynamically selects the most suitable reward model for each query, based on task metadata, and synthesizes new reward tools when existing ones are inadequate. This process is supported by the Group Relative Policy Optimization (GRPO) algorithm, which improves training stability and efficiency by estimating advantages relative to output groups, reducing reliance on value models.
Experimental results across multiple benchmarks demonstrate RLAR's superiority over static reward baselines, with performance improvements of 10-60%. On the REWARDBENCH-V2 dataset, RLAR approaches the performance upper bound, significantly enhancing generalization, especially in out-of-distribution scenarios. It also reduces API token costs by 80% and GPU training hours by 75%, showcasing its scalability.
The framework's ability to self-adapt and expand reward tools makes it highly versatile, capable of handling diverse tasks without extensive domain expertise or costly annotations. Its robustness against reward hacking and format biases further underscores its practical value. Looking ahead, RLAR's approach opens new avenues for scalable, adaptive reinforcement learning, with potential extensions into multimodal reward systems and federated training, pushing the boundaries of AI alignment and multi-task learning.
Deep Dive
Abstract
Large language model alignment via reinforcement learning depends critically on reward function quality. However, static, domain-specific reward models are often costly to train and exhibit poor generalization in out-of-distribution scenarios encountered during RL iterations. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize programmatic verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. Experimental results demonstrate that RLAR yields consistent performance gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks. On RewardBench-V2, RLAR significantly outperforms static baselines and approaches the performance upper bound, demonstrating superior generalization through dynamic reward orchestration. The data and code are available on this link: https://github.com/ZhuoerFeng/RLAR.