Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing
WGRPO pairs rare successes and failures, raising AIME 2025 Pass@8 from 16.8 to 22.2.
Key Findings
Methodology
The paper introduces positive–negative prompt pairing with Weighted GRPO (WGRPO). Each update selects a hard-but-solvable q+ and an easy-but-brittle q−, generating rare successes and rare failures. For each prompt, G=8 rollouts are mapped to +1 for correct and −λneg for incorrect outcomes, then group-normalized. This amplifies informative tail events rather than relying on historical accuracy variance.
Key Results
- On Qwen2.5-Math-7B, two-prompt WGRPO raises AIME 2025 Pass@8 from 16.8 for the high-variance GRPO baseline to 22.2, and AMC23 Pass@64 from 94.0 to 97.0.
- Across MATH500, AIME 2025, and AMC23, the method is generally consistent and recovers much of the performance of GRPO trained on the 1,209-prompt DeepScaleR-sub pool; it exceeds the large-pool method on AMC23 at larger k.
- Ablations indicate that prompt pairing and WGRPO are complementary: replacing either the optimizer or the easy prompt weakens the result, showing that signal construction matters beyond prompt variance.
Significance
The work reframes low-data RLVR prompt selection as a problem of directional information, not merely difficulty or variance. Each update receives both a reliable positive demonstration and an explicit negative warning. This offers a compact, interpretable alternative to large prompt pools and human preference labels, with practical value for verifier-based reasoning systems operating under strict compute budgets.
Technical Contribution
WGRPO maps binary outcomes to signed weighted outcomes and applies group normalization. For empirical success rate p=k/G, correct and incorrect advantages are approximately A+=(1−p)/√p(1−p) and A−=−p/√p(1−p), so p≈1/G emphasizes successes while p≈1−1/G emphasizes failures. Pair selection uses lightweight on-policy probing to target both regimes, then optimizes a clipped GRPO objective plus βKL(πθ||πref).
Novelty
The central novelty is not GRPO itself or negative feedback alone, but deliberately constructing rare success and rare failure as a single bidirectional teaching unit. Unlike historical-variance heuristics, the method directly targets the geometry and information density of the update, producing interpretable “do” and “do-not” signals.
Limitations
- The evidence is limited to Qwen2.5-Math-7B, its Instruct variant, mathematical datasets, and deterministic binary verification. It does not establish equivalent gains for open-ended generation, soft rewards, larger models, or substantially different task distributions.
- The method depends on probing and a suitable candidate pool. If estimates are inaccurate or prompts become all-correct or all-wrong, within-group variance collapses and the intended amplification can disappear; fixed prompts may also become mismatched as the policy improves.
Future Work
Future work should dynamically refresh q+ and q− as policy competence changes, extend the method to code, formal proof, and tool-use tasks, and analyze the roles of G, λneg, εstd, probing budget, and KL regularization. Larger models, longer training, and non-binary verifiers are important tests of robustness.
AI Executive Summary
Reinforcement learning with verifiable rewards (RLVR) has become a powerful way to improve mathematical reasoning without human preference labels or a learned reward model. Yet in the extreme low-data regime, the choice of training prompts remains poorly understood. Existing approaches often select prompts with high historical accuracy variance. Such prompts may expose uncertainty, but variance alone does not guarantee that a minibatch contains a clear learning direction; optimization can therefore be dominated by sampling noise.
Pang and colleagues propose positive–negative prompt pairing and Weighted GRPO (WGRPO). Each update uses only two prompts: a hard-but-solvable q+, where rare correct answers provide strong positive anchors, and an easy-but-brittle q−, where rare failures provide sharp negative guidance. WGRPO samples eight responses per prompt, assigns weighted signed outcomes, and normalizes them within each group. A success rate near 1/8 therefore amplifies rare successes, whereas a rate near 7/8 amplifies rare failures. Lightweight probing selects candidates from AIME 2025 and DeepScaleR-sub.
On Qwen2.5-Math-7B, the two-prompt method improves AIME 2025 Pass@8 from 16.8 for a high-variance GRPO baseline to 22.2, and AMC23 Pass@64 from 94.0 to 97.0. It remains competitive with GRPO trained on 1,209 DeepScaleR-sub prompts and shows similar gains on Qwen2.5-Math-7B-Instruct. The broader lesson is that useful RLVR data is defined not only by quantity or variance, but by whether each update contains informative positive and negative evidence. The main caveats are limited task diversity, fixed prompt selection, and dependence on deterministic verification.
Deep Analysis
Background
RLVR trains language models with deterministic verifiers such as exact-answer checking, avoiding human preference annotation and a separate reward model. PPO- and GRPO-style methods improve optimization under sparse rewards. LIMR shows that prompt sets can be compressed, while one-shot RLVR reports gains from high-variance prompts. What remains unclear is which mechanisms make an extremely small prompt set informative and transferable.
Core Problem
With very few prompts, binary rewards yield sparse evidence. A hard problem may produce an occasional success, while an easy problem may produce an occasional failure. If these tail events are absent from a minibatch, the gradient direction becomes unstable. Historical accuracy variance measures fluctuation but cannot distinguish a useful positive anchor from a useful negative warning.
Innovation
- �� Positive–negative pairing: q+ supplies rare successes and q− supplies rare failures.
- �� WGRPO: weighted signed outcomes plus group normalization amplify the relevant tail event.
- �� Lightweight probing: candidates from AIME 2025 and DeepScaleR-sub are matched to approximately 1/8 and 7/8 success rates.
- �� Minimal training: two fixed prompts can approach a 1,209-prompt RLVR baseline.
Methodology
- �� Sample G=8 responses for each prompt and compute p=k/G.
- �� Map each outcome to yi=+1 if correct, otherwise −λneg.
- �� Compute group mean μq and standard deviation σq, then use AWGRPO=(yi−μq)/(σq+εstd), broadcast over valid tokens.
- �� Insert the advantage into the clipped GRPO objective and add βKL(πθ||πref).
- �� Select q+ with p≈1/8 and q− with p≈7/8; the first reinforces rare correct trajectories, while the second penalizes rare failures.
- �� Train with B=2 prompts per update, keeping the pair fixed after probing.
Experiments
The study trains Qwen2.5-Math-7B and Qwen2.5-Math-7B-Instruct using AIME 2025 and DeepScaleR-sub, the latter containing 1,209 training prompts. Each update uses B=2 and G=8, for up to 500 steps. Baselines include GRPO on the full DeepScaleR-sub pool and GRPO on two historically high-variance prompts. Evaluation uses AIME 2025, AMC23, and MATH500, temperature 0.6, maximum length 3,072, and Pass@k for k=1–64. Ablations replace WGRPO or the easy prompt.
Results
WGRPO with easy–hard pairing consistently beats the two-high-variance-prompt GRPO baseline. AIME 2025 Pass@8 rises from 16.8 to 22.2, while AMC23 Pass@64 rises from 94.0 to 97.0. Gains on MATH500 are smaller but generally consistent. The method remains competitive with 1,209-prompt GRPO and surpasses it on AMC23 at larger k. Ablations support the need for both the optimizer and the paired prompt regimes.
Applications
The approach is suited to mathematics, unit-tested code generation, formal proof, and executable planning—tasks with reliable verifiers. A practitioner needs a candidate pool, a modest probing budget, and balanced positive and negative prompts. It may reduce post-training compute while preserving exploration, but deployment requires monitoring verifier quality, distribution shift, and degenerate rollout groups.
Limitations & Outlook
The study covers two related 7B models, fixed two-prompt training, and deterministic binary rewards. As training changes the policy, fixed q+ and q− may leave their target regimes; probing error can also produce all-correct or all-wrong groups. The roles of λneg, G, εstd, KL strength, and training duration are not fully characterized. Future work should use adaptive pairing, broader tasks, larger models, soft rewards, and direct compute–quality analyses.
Plain Language Accessible to non-experts
Imagine a teacher training a student with only two practice cards. The old teacher chooses cards where the student’s results fluctuate the most, but that does not always reveal a useful lesson. The new teacher deliberately chooses one very difficult card that the student can occasionally solve and one easy card that the student occasionally gets wrong.
The student tries each card eight times. If the hard card is solved only once, that rare success receives special attention: “Remember this route; it works.” If the easy card is solved seven times but missed once, the mistake also receives special attention: “Find this trap and avoid it.” Each lesson therefore contains both encouragement and correction.
This is WGRPO. It changes the weight of each result according to how unusual it is within the eight attempts. With Qwen2.5-Math-7B, just two such cards raised AIME 2025 Pass@8 from 16.8 to 22.2 and AMC23 Pass@64 from 94.0 to 97.0. The lesson is simple: more practice cards are not always better; carefully chosen successes and failures can teach more clearly.
ELI14 Explained like you're 14
Think of an AI playing a maths game. The coach has only two training levels. Level one is tough, but the AI sometimes wins. Level two is easy, but the AI occasionally messes up. Instead of ignoring these unusual moments, the new method treats them as the best clues: a rare win says, “Keep using that strategy,” while a rare loss says, “Watch out for this trap!”
The AI plays each level eight times. If it gets the hard level right once, that success gets extra encouragement. If it gets the easy level wrong once, that failure gets extra correction. So every training round teaches both what to do and what not to do. That is why the method is called Weighted GRPO: outcomes receive different weights before the model learns from them.
The results are impressive for such a tiny training set. On Qwen2.5-Math-7B, AIME 2025 Pass@8 rises from 16.8 to 22.2, and AMC23 Pass@64 rises from 94.0 to 97.0. It is like improving at a game by studying one surprising victory and one silly mistake. Pretty efficient, right? The catch is that the method has mainly been tested on maths, so researchers still need to check whether it works for coding and other challenges.
Glossary
RLVR (Reinforcement Learning with Verifiable Rewards)
A reinforcement-learning paradigm in which an automatic verifier supplies the reward. For mathematics, the reward is commonly binary correctness under exact-answer checking.
The overall training framework studied in the paper.
GRPO (Group Relative Policy Optimization)
A policy-optimization method that samples several responses to the same prompt and derives relative advantages from their group rewards. It avoids requiring a conventional value model in the usual formulation.
The baseline optimizer and foundation for WGRPO.
WGRPO (Weighted GRPO)
A GRPO variant that converts binary outcomes into weighted signed outcomes before group normalization. This automatically emphasizes rare successes or rare failures.
The paper’s main optimization contribution.
Positive–negative pairing
Selecting a hard-but-solvable prompt and an easy-but-brittle prompt together. The former supplies a positive anchor; the latter supplies a negative warning.
The prompt-selection strategy used at every update.
Pass@k
The probability that at least one of k generated answers is correct. Larger k measures whether the model can explore a latent correct solution.
The principal metric on AIME 2025, AMC23, and MATH500.
Rare-event amplification
A group-normalization effect that gives large positive weight to uncommon successes in low-success groups and large negative weight to uncommon failures in high-success groups.
The mechanism explaining WGRPO’s advantage geometry.
Open Questions Unanswered questions from this research
- 1 Adaptive pairing remains unresolved: fixed prompts may leave the target 1/8 or 7/8 regimes as the policy improves, requiring reliable online replacement without destabilizing training.
- 2 Cross-task generalization is unknown. Code, formal proof, and soft verifiers may violate the binary-outcome assumptions behind the closed-form advantage analysis.
Applications
Immediate Applications
Low-data mathematical post-training
A model team can probe candidate problems, select one near 1/8 success and one near 7/8 success, then train with G=8 WGRPO rollouts. The method is suitable for compute-limited experiments and can be evaluated directly with Pass@k.
Verifier-guided code generation
Unit-test outcomes can define success and failure. Teams could pair difficult programs that occasionally pass with easy programs that occasionally fail, provided tests are reliable and failures reflect meaningful model behavior rather than ambiguous specifications.
Long-term Vision
Adaptive capability curricula
A future system could continually monitor prompt success rates and replace q+ and q− as competence changes. Such a curriculum could reduce data requirements while balancing exploration, correction, and transfer across tasks.
Abstract
Reinforcement learning with verifiable rewards (RLVR) is effective for training large language models on deterministic outcome reasoning tasks. Prior work shows RLVR works with few prompts, but prompt selection is often based only on training-accuracy variance, leading to unstable optimization directions and weaker transfer. We revisit prompt selection from a mechanism-level view and argue that an effective minibatch should provide both (i) a reliable positive anchor and (ii) explicit negative learning signals from rare failures. Based on this principle, we propose \emph{positive--negative pairing}: at each update, we sample a hard-but-solvable $q^{+}$ and an easy-but-brittle prompt $q^{-}$(high success rate but not perfect), characterized by low and high empirical success rates under multiple rollouts. We further introduce Weighted GRPO, which reweights binary outcomes at the pair level and uses group-normalized advantages to amplify rare successes on $q^{+}$ into sharp positive guidance while turning rare failures on $q^{-}$ into strong negative penalties. This bidirectional signal provides informative learning feedback for both successes and failures, improving sample efficiency without suppressing exploration. On Qwen2.5-Math-7B, a single paired minibatch per update consistently outperforms a GRPO baseline that selects two prompts via commonly used variance-based selection heuristics: AIME~2025 Pass@8 improves from 16.8 to 22.2, and AMC23 Pass@64 from 94.0 to 97.0, while remaining competitive with large-scale RLVR trained from a pool of 1209 training prompts. Similar gains are observed on Qwen2.5-Math-7B-Instruct.