RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

The evolution of reinforcement learning in NLP has seen methods like REINFORCE and PPO dominate, relying on explicit reward signals. GRPO further improved efficiency by normalizing rewards within groups, making training more stable for verifiable tasks like GSM8K. However, many real-world NLP tasks—such as dialogue summarization, long-form explanation, and open-ended generation—lack clear success signals, often involving subjective judgments. Recent advances like RLHF and preference learning attempt to address this by using human feedback or relative preferences, but they still depend on some form of explicit supervision. The challenge remains: how to effectively optimize policies when success is ambiguous or hard to verify, especially in large-scale, diverse tasks. This paper situates itself within this context, proposing a method that leverages weak supervision via contrastive learning to overcome these limitations.

The core issue is the reliance on verifiable rewards, which are often unavailable or unreliable in many NLP tasks. Without explicit success signals, existing reinforcement learning approaches struggle with stability and scalability. The key bottleneck is how to construct meaningful advantage signals that guide policy improvement without explicit correctness labels. Pseudo-target methods and preference-based signals provide some solutions but are often brittle or noisy. The challenge is to develop a unified framework that can utilize weak, instance-specific cues to generate stable, informative advantages, enabling effective policy updates across diverse, subjective, and complex tasks.

RRPO introduces a paradigm shift by replacing explicit correctness rewards with reference-relative contrastive advantages. Its main innovation is the generation of positive and negative anchor sets using stratified conditional rollouts guided by weak task conditions. An offline-trained contrastive metric compares candidate rollouts against these anchors, producing alignment scores that reflect relative desirability. During policy training, the projection head is frozen, and the normalized contrastive scores serve as advantages, enabling stable, group-relative updates. This approach combines the strengths of contrastive learning, weak supervision, and group normalization, providing a scalable and robust alternative to traditional reward-based methods.

• �� Anchor Generation: Use weak task-specific conditions to generate positive and negative rollout sets via a fixed anchor policy. • Offline Metric Learning: Train a contrastive model with set-contrastive loss to distinguish positive and negative anchors, creating a similarity space. • Advantage Construction: During policy updates, compute alignment scores between rollouts and anchors, freeze the metric head, and normalize scores within each group. • Policy Optimization: Apply a clipped policy gradient objective, replacing scalar rewards with normalized contrastive advantages. • Training Workflow: First, offline train the contrastive metric; then, perform policy optimization with frozen metric, ensuring stability and robustness.

Experiments cover verifiable reasoning (GSM8K), open-ended generation (ELI5, SAMSum), and post-fine-tuning scenarios. Multiple model sizes (Qwen2.5-1.5B, Qwen3-1.7B, Qwen3-4B) are evaluated. Baselines include standard instruction tuning, verifier-based Dr.GRPO, and pseudo-target fine-tuning. The evaluation metrics include Pass@k, ROUGE, BLEU, and BERTScore. Anchor sets are generated with weak conditions, and the offline contrastive model is trained on initial rollouts. Policy updates use standardized contrastive scores as advantages, with ablation studies confirming the importance of anchor quality and model parameters.

RRPO consistently outperforms baseline instruction models, achieving Pass@1 of 0.768 on GSM8K with Qwen3-1.7B, surpassing verifier-based methods. In open-ended tasks, RRPO improves ROUGE-L by 2-3 points over pseudo-target fine-tuning, with stable training curves. Post-fine-tuning, RRPO further enhances ROUGE scores by 1-2 points, demonstrating its complementary nature. These results confirm that contrastive, reference-relative advantages are effective across diverse tasks, especially where explicit verification is infeasible.

RRPO is suitable for tasks lacking explicit success signals, such as long-form content creation, dialogue summarization, and preference-based learning. It can be integrated into large language model training pipelines to improve stability and generalization, reducing reliance on costly annotations. Its adaptability makes it promising for multi-modal applications, multi-turn dialogues, and real-time systems where explicit verification is impractical.

The quality of anchor sets critically impacts performance; poor anchors can introduce bias. Offline contrastive training adds computational overhead, limiting scalability. In highly noisy or ambiguous environments, the contrastive advantage may become unstable, requiring further robustness enhancements. Future work should focus on adaptive anchor generation and online contrastive learning to address these issues.