CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning
CoRE uses graph-based dominant set extraction and replicator dynamics to improve test-time RL rewards, boosting accuracy by 21.7 points.
Key Findings
Methodology
CoRE constructs a graph from N sampled trajectories, where edges encode answer agreement, reasoning similarity, and confidence. Using replicator dynamics, it extracts a dominant set representing mutually supportive trajectories, producing refined pseudo-labels, graded per-trajectory rewards, and a question-level cohesiveness gate. This framework generalizes majority voting, with theoretical analysis showing thresholds for minority support recovery, enhanced by confidence calibration. Experiments across seven models and five benchmarks demonstrate an average gain of 21.7 points over untrained baselines, outperforming traditional voting, especially in contested agreement scenarios, while reducing training steps by 54-70%. The approach leverages multi-source signals for robust self-supervised rewards without additional roll-outs.
Key Results
- Across 42 model-benchmark pairs, CoRE improves accuracy by +21.7 points on average, surpassing majority voting (+20.4), with up to +7.5 points in ambiguous cases. It reaches baseline accuracy 54-70% faster, confirming theoretical predictions. Ablation studies show the synergy of graph structure and confidence calibration is essential, with each component alone insufficient to outperform voting. The method excels in complex reasoning tasks, especially where minority support is critical, demonstrating robustness across diverse models and datasets.
Significance
This work advances self-supervised reinforcement learning by transforming trajectory relationships into a graph, enabling extraction of mutually supportive groups via dominant set theory. It addresses the core challenge of recovering minority correct trajectories in the presence of dominant wrong answers, crucial for improving reasoning accuracy in large language models. The theoretical insights into thresholds for minority support and confidence calibration provide a rigorous foundation for future model fusion and consensus strategies. Practically, CoRE enhances model robustness and efficiency, reducing training steps significantly, and opens pathways for more reliable autonomous reasoning in AI systems, with broad implications for AI safety, interpretability, and deployment in complex environments.
Technical Contribution
This paper pioneers the application of dominant set extraction via replicator dynamics to reinforcement learning reward design, integrating answer agreement, reasoning similarity, and confidence into a unified graph framework. It provides a rigorous theoretical analysis of the conditions under which minority correct trajectories can overturn majority errors, with explicit thresholds modulated by confidence calibration. The method generalizes voting, allowing graded rewards and question-level gating, all without auxiliary models or additional roll-outs. Its integration into policy gradient methods like GRPO demonstrates a novel synergy between combinatorial optimization, dynamical systems, and reinforcement learning, offering a new paradigm for self-supervised reward construction.
Novelty
The key novelty lies in modeling multiple model trajectories as a weighted graph and applying dominant set extraction through replicator dynamics to derive refined pseudo-labels and rewards. Unlike traditional voting or confidence-weighted schemes, CoRE captures relational support among trajectories, enabling recovery of minority correct solutions even under strong class imbalance. The theoretical analysis of thresholds for support recovery, combined with confidence calibration, provides a new understanding of consensus formation in model ensembles, marking a significant departure from prior work relying solely on count-based voting or heuristic weighting.
Limitations
- The effectiveness of CoRE depends on accurate confidence calibration; overconfident incorrect trajectories can mislead the support extraction. In highly noisy or imbalanced scenarios, recovery of minority support remains challenging.
- Computational complexity increases with the size of the trajectory graph, potentially limiting real-time application in large-scale settings.
- The method assumes the availability of meaningful confidence scores and relational signals; in cases where these are unreliable, performance may degrade. Further work is needed to improve scalability and robustness.
Future Work
Future directions include optimizing graph construction and dominant set extraction algorithms for scalability, integrating multi-modal signals, and extending the framework to multi-task and multi-modal environments. Additionally, refining theoretical bounds for support recovery under various noise conditions and exploring adaptive parameter tuning could enhance robustness. Combining CoRE with other self-supervised learning paradigms and applying it to real-world applications such as autonomous reasoning systems, dialogue, and multimodal AI are promising avenues.
AI Executive Summary
In the rapidly evolving field of natural language processing, large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, their performance in unlabeled test environments remains limited by reliance on simplistic aggregation methods like majority voting. Traditional test-time reinforcement learning (TTRL) employs majority vote to generate pseudo-labels, rewarding trajectories that match the most common answer. While effective in many cases, this approach falters when minority correct solutions are overshadowed by incorrect but more frequent answers, leading to systematic errors and degraded model robustness.
Addressing this challenge, the paper introduces CoRE (Consensus Rewards via Equilibrium), a novel framework that transforms the set of sampled trajectories into a graph structure. Edges encode answer agreement, reasoning similarity, and generation confidence, capturing the relational support among trajectories. Using replicator dynamics—a process inspired by evolutionary game theory—CoRE extracts the dominant set of mutually supportive trajectories, which serve as a refined pseudo-label and graded reward signals. This approach generalizes the traditional voting mechanism, with theoretical analysis revealing the conditions under which minority correct solutions can overturn majority errors. Confidence calibration further enhances support recovery by exponentially lowering the threshold needed for minority support.
Empirical results across seven model architectures and five benchmark datasets demonstrate the effectiveness of CoRE. On average, it improves accuracy by 21.7 points over untrained baselines, surpassing the 20.4 points achieved by standard majority voting. Notably, in scenarios with high disagreement, CoRE outperforms voting by up to 7.5 points and reaches baseline accuracy 54-70% faster in training steps. These findings confirm that modeling trajectory relationships as a graph and extracting mutual support offers a robust, self-supervised reward signal that enhances model reasoning, especially in complex, minority-supported cases.
The broader impact of this work lies in its potential to improve autonomous reasoning systems, reduce reliance on labeled data, and foster more reliable AI in real-world applications. By bridging combinatorial optimization, dynamical systems, and reinforcement learning, CoRE opens new avenues for scalable, interpretable, and robust AI training paradigms. Future research may focus on algorithmic efficiency, multi-modal extensions, and deployment in dynamic environments, further advancing the frontier of self-supervised AI learning.
Deep Dive
Abstract
On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.