Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Reasoning Arena improves reasoning models via adaptive routing and trace tournaments, achieving a 7.6% average performance boost.
Key Findings
Methodology
Reasoning Arena combines verifiable rewards with LLM-based judge systems. Non-diverse reward groups are routed to trace tournaments, where reasoning traces are compared. Bradley-Terry models fit incomplete comparison graphs to reduce computational complexity.
Key Results
- Achieved a 7.6% average improvement across reasoning tasks, including a 12.9% boost on AIME 2026.
- Training efficiency improved by 27%-41%, with nearly 50% reduction in generation compute.
- Extracted gradient signals from non-diverse groups, significantly enhancing reasoning performance.
Significance
Addresses RLVR's core bottleneck of wasted compute in non-diverse reward groups. The framework improves both performance and efficiency, offering impactful advancements for academia and industry.
Technical Contribution
Introduces trace tournaments as a reward mechanism for RLVR, paired with adaptive routing and Bradley-Terry models to achieve scalable reward estimation.
Novelty
First to integrate trace tournaments into RLVR for reasoning tasks, dynamically routing non-diverse groups to extract fine-grained reward signals.
Limitations
- Relies on LLM judge quality, which may introduce biases.
- Dynamic routing adds computational overhead.
- May not fully replace traditional reward mechanisms in certain tasks.
Future Work
Future research may explore more efficient judge models, improved routing strategies, and broader applications across reasoning domains.
AI Executive Summary
Reinforcement learning with verifiable rewards (RLVR) has been a dominant method for improving reasoning in large language models. However, RLVR struggles with non-diverse reward groups, where all traces receive identical rewards, leading to wasted compute and no gradient signal. Reasoning Arena addresses this by dynamically routing such groups to trace tournaments, where reasoning quality is compared to generate fine-grained reward signals.
Experiments show that Reasoning Arena achieves a 7.6% average improvement across tasks like competition mathematics and code generation, with a 12.9% boost on AIME 2026. It also accelerates training by 27%-41% and reduces generation compute by nearly 50%. Using Bradley-Terry models to fit incomplete comparison graphs, the framework avoids quadratic complexity.
This research not only resolves RLVR's bottlenecks but also introduces a novel reward mechanism for reasoning tasks. Future work could explore more efficient judge systems and expand applications to other domains, offering transformative potential for academia and industry alike.
Deep Analysis
Background
Large language models have shown remarkable reasoning capabilities, with RLVR emerging as a key training paradigm. However, RLVR fails to provide gradient signals for non-diverse reward groups, wasting compute and limiting optimization potential.
Core Problem
RLVR relies on reward variance within groups to generate gradients. Non-diverse groups, where all traces receive identical rewards, result in zero advantage and wasted compute. This issue is prevalent in both early and late training stages.
Innovation
Reasoning Arena introduces adaptive routing to detect non-diverse groups and assigns them to trace tournaments. These tournaments compare reasoning trajectories to generate granular reward signals. Bradley-Terry models fit incomplete comparison graphs, reducing computational complexity.
Methodology
- �� Adaptive routing: Routes groups based on reward variance.
- �� Trace tournaments: Compares reasoning trajectories within groups.
- �� Bradley-Terry model: Fits incomplete comparison graphs for reward estimation.
- �� Experimental validation: Evaluates performance on reasoning tasks like AIME and LiveCodeBench.
Experiments
Experiments use Ministral-3-8B-Instruct as the policy model, evaluating tasks like AIME mathematics and LiveCodeBench coding. Baselines include RLVR, RLAIF, and ArenaRL, with metrics like average pass rates and training efficiency.
Results
Reasoning Arena achieves a 7.6% average improvement, with a 12.9% boost on AIME 2026. Training efficiency improves by 27%-41%, and generation compute reduces by nearly 50%. The framework effectively extracts gradient signals from non-diverse groups.
Applications
Applicable to reasoning tasks like competition mathematics and code generation, particularly in scenarios requiring granular reward signals, such as education and automated reasoning systems.
Limitations & Outlook
Relies on LLM judge quality, which may introduce biases. Dynamic routing adds computational overhead. May not fully replace traditional reward mechanisms in certain contexts.
Plain Language Accessible to non-experts
Imagine a cooking competition where judges only evaluate the final dishes but ignore the cooking process. Reasoning Arena acts like a new judge who watches every step — chopping, seasoning, plating — and scores based on the process. Even if two dishes look identical, the better process earns higher marks.
ELI14 Explained like you're 14
Imagine playing a video game where everyone scores the same, but you feel your moves were cooler. Reasoning Arena is like a new judge who watches your gameplay and gives extra points for style! Now it's not just about the score but how you got there — fun, right?
Glossary
RLVR (Reinforcement Learning with Verifiable Rewards)
A method using rule-based rewards to train reasoning models.
Used to generate group-relative reward signals for optimization.
Bradley-Terry Model
A statistical model estimating object strengths from comparison data.
Fits incomplete graphs in trace tournaments.
Trace Tournament
A method comparing reasoning trajectories to generate reward signals.
Applied to non-diverse reward groups.
Adaptive Routing
A strategy selecting reward mechanisms based on group diversity.
Separates diverse and non-diverse groups.
Non-Diverse Reward Group
Groups where all traces receive identical rewards.
A key bottleneck in RLVR training.
Open Questions Unanswered questions from this research
- 1 How to further improve LLM judge quality to reduce biases?
- 2 Can this framework be extended to non-reasoning tasks?
Applications
Immediate Applications
Math Competition Optimization
Helps models solve complex math problems and improve competition performance.
Code Generation
Enhances reasoning in code generation models, reducing errors.
Long-term Vision
Automated Education Systems
Develop reasoning-optimized tools for better student learning outcomes.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models through outcome-based supervision. However, verifiable rewards frequently become uninformative at the group level: when all sampled traces of a given prompt receive identical rewards, group-relative advantage estimation provides no gradient signal, even though the traces may differ substantially in reasoning quality. We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them. Beyond examining the final answer, Reasoning Arena constructs trace tournaments, where reasoning traces are compared head-to-head to expose finer-grained preferences within the group, converting reasoning quality into rich relative reward signals. To make reward estimation efficient, rather than exhaustively comparing every pair, each new trace is evaluated against a small, dynamically updated pool of previously generated traces as anchors to efficiently establish a relative ranking. We then fit a Bradley-Terry model on the incomplete comparison graph, enabling scalable RL integration without quadratic pairwise comparisons. Empirical results demonstrate that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks. By converting otherwise wasted zero-advantage samples into useful gradient updates, our method accelerates training by 27% to 41%, saving nearly 50% of generation compute, and substantially improves overall reasoning performance.