Reward-Free Evolving Agents via Pairwise Validator
Proposes pairwise validator as reward substitute in self-evolving agents, matching or surpassing full reward performance with reduced labeling cost.
Key Findings
Methodology
This paper introduces a pairwise validator based on frozen LLMs to replace scalar rewards in self-evolving loops. The validator compares parent and child candidates, providing a binary decision on which is better, thus avoiding the calibration issues of absolute scoring. Two variants are proposed: Adaptive Focus, which retains existing validation set parent selection, and Soft Elo, which uses validator verdicts to drive parent selection, eliminating rewards entirely. Integration into systems like GEPA, ADRS, and ShinkaEvolve demonstrates that the approach matches or exceeds baseline performance across multiple tasks and data sources, with robustness to validator swaps across models.
Key Results
- Across ten diverse tasks (retrieval, instruction-following, math, code evolution), the methods achieved performance comparable or superior to full reward baselines, especially in low-data regimes (validation set of 12-30 examples). For instance, Qwen3-8B on HotpotQA reached 78.5% accuracy, surpassing baseline results. In math tasks, accuracy improved from 40% to 60% with few-shot reward-free methods, indicating effective substitution of reward signals.
- The approach consistently reduced validation-test gaps, indicating better generalization. On tasks like HoVer and IFBench, Soft Elo and Adaptive Focus outperformed full reward methods, demonstrating robustness against overfitting to validation data.
- Ablation studies confirmed that replacing scalar rewards with pairwise judgments maintains or enhances task performance, with minimal additional computational cost, and works across different models and task types.
Significance
This work addresses the core challenge of reward design in autonomous systems, significantly lowering annotation costs and enabling scalable self-improvement. By leveraging contrastive judgments from frozen LLMs, it offers a practical, effective alternative to traditional reward signals, facilitating broader deployment of self-evolving agents in real-world applications. The approach enhances robustness, reduces reliance on labeled data, and opens new avenues for unsupervised and semi-supervised learning frameworks, impacting both academia and industry.
Technical Contribution
The main technical innovation lies in integrating a frozen LLM as a pairwise validator within self-evolving loops, replacing explicit scalar rewards. This enables training-free, contrastive judgments that are more stable and easier to calibrate. The two variants—Adaptive Focus and Soft Elo—offer flexible mechanisms for parent selection and candidate evaluation, with theoretical grounding in Elo rating systems and heuristic rubric updates. The method's simplicity and effectiveness extend the scope of reward-free optimization, providing a new paradigm for autonomous model refinement.
Novelty
This is the first work to embed a pairwise, zero-shot LLM-based validator directly into the per-iteration accept/reject step of self-evolving agents, replacing scalar reward signals. Unlike prior preference learning or reward modeling, this approach requires no additional training, making it highly scalable and adaptable. Its contrastive judgment mechanism offers superior stability and generalization, representing a significant step forward in reward-free reinforcement and evolutionary learning.
Limitations
- The quality of pairwise judgments depends on the representativeness of the examples and the judgment capability of the frozen LLM, which may falter in highly ambiguous or complex tasks.
- Current experiments are limited to small validation pools (12-30 examples), and scalability to larger, more diverse datasets remains to be validated.
- Fully reward-free variants like Soft Elo may suffer from slower convergence or performance variability in certain tasks, requiring further optimization of update rules and judgment criteria.
Future Work
Future research could explore multi-modal and multi-objective pairwise validators, adaptive sampling strategies, and dynamic rubric updates to improve judgment robustness. Combining contrastive evaluation with reinforcement learning principles may further enhance autonomous self-improvement. Extending the approach to more complex, real-world scenarios with multiple conflicting objectives and larger datasets will be crucial for practical deployment.
AI Executive Summary
In recent years, the development of autonomous, self-evolving AI agents has gained significant attention. These systems iteratively propose modifications—such as prompt templates or code snippets—and evaluate their quality to guide continuous improvement. Traditionally, this process relies heavily on scalar reward signals, which are costly to design and annotate, especially for complex tasks. This paper introduces a novel approach that replaces explicit scalar rewards with a pairwise validator based on frozen large language models (LLMs). The validator compares parent and candidate solutions, providing a binary judgment on which is better, effectively sidestepping the challenges of reward calibration and labeling costs.
The authors integrate this pairwise validator into existing self-evolving frameworks like GEPA, ADRS, and ShinkaEvolve, developing two variants: Adaptive Focus, which retains the original validation set parent selection, and Soft Elo, which uses validator verdicts to drive parent selection via an Elo rating system. Extensive experiments across ten diverse tasks—including retrieval, instruction-following, mathematical reasoning, and code evolution—demonstrate that these reward-free methods match or outperform the traditional full-reward baselines in most settings. Notably, in low-data regimes, the methods show improved generalization, with reduced validation-test gaps, indicating better robustness.
The core innovation lies in leveraging contrastive judgments from frozen LLMs, which are training-free and require minimal additional computation. This approach simplifies reward design, reduces annotation costs, and enhances scalability. The results suggest that pairwise comparison mechanisms can serve as effective substitutes for scalar rewards in autonomous systems, opening new avenues for scalable, label-efficient AI development.
Despite promising results, limitations include dependence on the judgment quality of the frozen LLM and challenges in scaling to larger, more complex datasets. Future work may focus on multi-modal, multi-objective judgments, adaptive sampling, and broader real-world applications. Overall, this work marks a significant step toward reward-free, self-improving AI agents with broad implications for research and industry.
Deep Dive
Abstract
A self-evolving agentic loop repeatedly proposes a tweaked version of an agent (its prompt template or program) and accepts or rejects the change based on a per-iteration quality signal. Designing that signal is often the costly part of the project: a reliable scalar reward requires domain expertise and labeled examples that are themselves as expensive to assemble as the agent's underlying task. We propose replacing the scalar at the accept/reject gate with a pairwise validator: a frozen LLM that, given the parent and child candidate, returns a binary verdict on which is better. Pairwise judgment is generally easier and more stable than absolute scoring, due to its contrastive nature, which mitigates the need for strict scale calibration. The validator also requires no training of its own. We integrate the validator into three published self-evolving engines (GEPA, ADRS, ShinkaEvolve) and report two flavors: Adaptive Focus, which retains the engine's existing val-set parent selection, and Soft Elo, which lets the validator's verdicts drive parent selection so that val-set rewards drop as well. Across multiple agents and two artifact substrates (prompt and code), our method matches or exceeds the full-reward baseline on the majority of settings we evaluate, and the pattern survives a cross-family validator swap. The pairwise gate is thus a drop-in replacement for per-step reward design at competitive task accuracy without the labeling cost.