Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
GOLF leverages group-level natural language feedback to boost RL exploration, achieving 2.2× sample efficiency.
Key Findings
Methodology
GOLF integrates external critiques and intra-group attempts via a feedback aggregation mechanism to generate high-quality refinements. It employs three core components: 1) Feedback aggregation consolidates diverse failure signals into a unified refinement prompt; 2) Adaptive injection introduces top-quality off-policy refinements in low-reward regions to mitigate exploration bottlenecks; 3) Joint optimization trains generation and refinement within a unified RL loop, creating a positive feedback cycle. The approach builds upon GRPO, leveraging multi-source feedback to enhance exploration diversity and efficiency. Extensive experiments on verifiable and non-verifiable tasks demonstrate that GOLF outperforms baseline RL methods, with a 2.2× increase in sample efficiency and significant performance gains across benchmarks.
Key Results
- On five non-verifiable benchmarks, GOLF achieves an average performance improvement of 22.7%, with a 2.2× sample efficiency gain. In AlpacaEval-v2.0, training steps are reduced from 180 to 80, a 2.25× efficiency boost. On WildBench and ArenaHard-v2.0, performance increases by 85.2% and 70.7%, respectively.
- In verifiable tasks, GOLF surpasses traditional RL and fine-tuning methods, improving scores such as AIME24 by +6.46 points and Pass@4 metrics in code generation. It demonstrates higher solution diversity and robustness, with consistent gains across math, instruction, and coding benchmarks.
- Ablation studies confirm that combining external critiques and intra-group attempts yields synergistic benefits. Adaptive off-policy injection is crucial in low-reward regimes, effectively alleviating gradient vanishing and accelerating learning.
Significance
This work addresses the fundamental limitation of RL relying solely on sparse scalar rewards. By exploiting rich natural language feedback, it significantly enhances exploration efficiency and model capabilities. The framework offers a new paradigm for autonomous learning in complex environments, enabling models to identify errors, explore diverse solutions, and learn faster. Its innovative feedback fusion mechanism paves the way for more intelligent, adaptable AI systems, with broad implications for natural language understanding, reasoning, and beyond.
Technical Contribution
GOLF introduces a novel multi-source feedback fusion framework, combining external critiques and intra-group attempts to generate high-quality refinements. It employs an adaptive injection strategy to incorporate these refinements as off-policy scaffolds, alleviating exploration bottlenecks. The approach extends the GRPO algorithm with mixed policy optimization, jointly training generation and refinement capabilities within a unified RL loop. Theoretical and empirical results demonstrate improved exploration diversity, sample efficiency, and model robustness, representing a significant advancement over existing methods.
Novelty
This is the first systematic integration of group-level natural language feedback, combining external critiques and intra-group attempts, with an adaptive off-policy injection mechanism. Unlike prior work that relies solely on scalar rewards or external critiques, GOLF leverages multi-source feedback to produce richer, more diverse exploration paths. Its joint optimization of generation and refinement within a unified RL framework marks a key innovation, enabling models to learn more efficiently and robustly in complex tasks.
Limitations
- The effectiveness heavily depends on the quality of natural language feedback; noisy or low-quality feedback can impair performance. Further research is needed to improve feedback filtering and robustness.
- In extremely sparse reward environments, the generation of effective correction samples remains challenging, potentially leading to exploration bias.
- Computational costs are high due to multi-source feedback processing and large model training, limiting scalability without substantial hardware resources.
Future Work
Future directions include integrating multi-modal feedback such as visual and auditory signals, enhancing feedback quality through better filtering and synthesis, and developing more efficient algorithms to reduce computational overhead. Additionally, exploring meta-learning and transfer learning strategies could further improve model adaptability and generalization across tasks and domains.
AI Executive Summary
Reinforcement learning (RL) has achieved notable success in training large language models (LLMs), yet it remains hampered by inefficient exploration due to reliance on sparse scalar rewards. Traditional RL approaches, such as PPO and GRPO, struggle to leverage the rich information embedded in natural language feedback, which can include explicit error diagnoses, comparative judgments, and concrete revision suggestions. This underutilization limits the models’ ability to explore diverse solutions and accelerates learning.
To address this challenge, the paper introduces GOLF (Group-level Natural Language Feedback), a novel RL framework designed to systematically incorporate multi-source natural language feedback into the training process. GOLF’s core innovation lies in aggregating external critiques and intra-group attempts to generate high-quality refinements. These refinements serve as off-policy scaffolds, injected adaptively into the training process to guide exploration in sparse-reward regions. The framework also jointly optimizes generation and refinement capabilities within a unified RL loop, creating a virtuous cycle that enhances both.
Experimental results across a broad spectrum of tasks—ranging from non-verifiable benchmarks like WildBench and ArenaHard to verifiable tasks such as mathematical reasoning, instruction following, and code generation—demonstrate GOLF’s effectiveness. The approach achieves a 2.2× improvement in sample efficiency, significantly reducing training steps while boosting performance. On non-verifiable tasks, GOLF outperforms baselines by 22.7%, and on verifiable tasks, it yields consistent accuracy gains, including a +6.46 point increase in AIME24 scores and improved Pass@k metrics.
These findings highlight the potential of leveraging rich natural language feedback to revolutionize RL training, making models more sample-efficient, diverse, and capable of tackling complex real-world problems. The research opens new avenues for multi-source feedback integration, adaptive exploration, and autonomous model refinement, with promising implications for future AI systems. Limitations include dependency on feedback quality and computational costs, guiding future work towards multi-modal feedback and efficiency improvements.
Deep Analysis
Background
近年来，深度学习与强化学习的结合推动了大规模预训练模型（如GPT系列、BERT）的快速发展。早期工作如RLHF（Ouyang et al., 2022）利用人类偏好进行微调，显著改善模型输出质量，但仍依赖稀疏的奖励信号，限制探索能力。随后，研究逐步引入自然语言反馈（NLF），如文本诊断、比较和修正建议（Ankner et al., 2024），试图提供更丰富的监督信息。Critique-GRPO（Zhang et al., 2025）等方法将外部批评融入RL，提升修正效率，但未充分利用多样化尝试和多源信息的潜力。探索策略的多样性和效率仍是瓶颈，尤其在奖励稀疏或梯度消失时，模型难以快速找到有效路径。本文提出的GOLF旨在融合多源自然语言反馈，突破现有限制，提升探索效率和模型能力。
Core Problem
核心问题在于传统RL方法对稀疏奖励的依赖导致探索效率低下，模型难以在复杂环境中快速找到高质量解。自然语言反馈中包含丰富的错误诊断和修正建议，但现有算法未能充分利用这些信息，导致探索路径受限，学习速度缓慢。特别是在奖励信号稀疏或梯度消失时，模型难以获得有效的训练信号，限制了其能力提升。如何有效融合多源、多层次的自然语言反馈，增强模型的探索能力，成为亟待解决的关键问题。
Innovation
GOLF的创新点在于：1）引入多源自然语言反馈的融合机制，将外部批评和组内尝试结合，生成高质量修正；2）设计自适应引入策略，在低奖励状态下动态注入修正样本，缓解梯度消失；3）在统一RL循环中联合优化生成和修正能力，形成正反馈机制。这些创新突破了以往仅依赖标量奖励的限制，极大丰富了模型的探索路径，提升了样本利用效率。算法在GRPO基础上扩展，结合多源反馈，有效增强模型的多样性和鲁棒性。
Methodology
- �� 采样响应组：对每个提示，生成N个响应，获得奖励和批评信息。• 反馈聚合：将失败响应及其批评整合成群组级修正提示，捕捉多样失败模式。• 生成修正：在聚合基础上，利用旧策略生成修正响应，评估质量。• 自适应引入：在低奖励状态下，从修正中随机引入高质量样本，替换失败响应，增强探索。• 联合优化：将生成和修正响应合并，使用混合目标（包括剪切的策略梯度）进行策略更新，确保模型在探索和修正能力上同步提升。
Experiments
采用五个非验证任务（如WildBench、ArenaHard）和验证任务（数学推理、指令跟随、代码生成）进行评估。模型包括Llama-3.1-8B和Qwen-3-8B，训练数据来自WildChat-IF和OpenR-Math等。对比基线包括微调和RL方法（如Refinement-FT、Critique-FT、GRPO、Critique-GRPO）。指标涵盖得分、Win Rate、Pass@k等。训练过程中，采用GPT-4作为评判标准，进行多轮采样和评估。关键超参数包括：采样组大小、奖励阈值、引入概率等。还进行了消融实验验证不同反馈源和引入策略的贡献。
Results
GOLF在五个非验证任务中平均性能优于最强基线22.7%，在样本效率方面提升2.2倍。训练步数由基线180步缩短至80步，效率提升超过2倍。数学推理得分提升6.46点，指令任务的Pass@4指标显著改善，整体表现优异。消融实验显示，外部批评和组内尝试互补性强，联合使用效果最佳。自适应引入机制在低奖励区域尤为关键，有效缓解梯度消失问题，显著提升探索效率和模型能力。
Applications
该方法可广泛应用于需要高效探索和修正的自然语言处理任务，如智能问答、自动编程、对话系统等。依赖自然语言反馈的融合机制，能在缺乏明确奖励信号的环境中提升模型性能。未来，结合多模态反馈（如视觉、语音）将进一步拓展应用场景，实现更智能的自主学习系统。
Limitations & Outlook
目前GOLF对高质量反馈的依赖较大，反馈噪声可能影响效果。在极端稀疏奖励环境中，修正样本的生成仍存在一定局限，可能导致探索偏差。模型训练和推理过程中的计算成本较高，尤其在大模型和多源反馈融合时，硬件资源需求显著增加。未来需优化反馈质量和引入机制，降低成本，增强鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做菜，遇到一道复杂的菜谱。传统做法就像只看食谱上的评分，知道菜做得好坏，但不知道怎么改进。而GOLF就像有一群厨友，他们会告诉你哪里做得不好（外部批评），还会尝试用不同的调料和方法（组内尝试），并把好的建议整理出来，告诉你下一次怎么做得更好。这样，你就能更快学会做出美味的菜，而且每次都能尝试不同的做法，变得越来越厉害。这种方法让你在厨房里变得更聪明、更快，做出更好吃的菜。
ELI14 Explained like you're 14
想象你在学校里学习新游戏，刚开始总是输，觉得很难。传统方法就像老师只告诉你你输了，没有告诉你怎么改进。而GOLF就像有一群朋友，他们会告诉你哪里出错了，还会试着用不同的方法玩游戏，帮你找到赢的方法。每次你失败后，他们会整理出一份建议，告诉你下一次怎么做得更好。这样，你学得更快，也能尝试很多不同的策略，不会一直卡在原地。就像和朋友一起玩游戏，大家互相帮忙，变得越来越厉害。
Glossary
Natural Language Feedback
Using human language to describe errors or suggestions, providing rich guidance. Technically, it involves the model understanding and utilizing diagnostic and revision information in text.
Refers to feedback generated by users or systems that guides policy improvement.
Group-level Feedback
Feedback aggregated from multiple responses, capturing diverse failure modes and partial ideas, offering comprehensive error signals.
Used to generate corrective suggestions and broaden exploration.
Off-policy scaffolds
High-quality corrections introduced as off-policy samples to assist exploration in sparse reward settings.
Alleviates gradient vanishing by injecting targeted guidance.
Mixed Policy Optimization
Combines on-policy and off-policy gradient updates to optimize the policy.
Enhances exploration diversity and learning efficiency.
Self-Refinement
Model’s ability to improve its outputs based on feedback, enhancing its correction and exploration capabilities.
Jointly trained to boost autonomous refinement.
Open Questions Unanswered questions from this research
- 1 如何在多模态、多源反馈中保持鲁棒性，减少噪声干扰，仍是未解难题。未来需研究更智能的过滤和融合机制，以实现更高效的自主学习。
Applications
Immediate Applications
Conversational AI
利用GOLF提升对话系统在用户反馈稀疏时的理解和生成能力，增强自主修正和探索能力。
Automated Coding Assistants
结合自然语言反馈优化代码生成模型，加快调试和修正流程，提升开发效率。
Long-term Vision
Autonomous Robots
未来机器人通过多源反馈自主探索复杂环境，提升自主决策和适应能力，逐步实现更智能的自主系统。
Abstract
Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2$\times$ improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.