Training Language Model to Critique for Better Refinement
RCO framework trains critic models via refinement signals, improving critique quality and response refinement by 15% on average.
Key Findings
Methodology
RCO employs a feedback loop where critic models generate critiques, guiding actor models to produce multiple refined responses. Critique Utility (CU) quantifies how much refinements are preferred over initial responses, serving as a reward for critic training. The process involves generating critiques, refinements, and evaluating preferences via a preference model, then optimizing critic parameters through a regularized objective that maximizes CU. This approach leverages scalar reward signals derived from response preferences, avoiding direct critique quality assessment, thus fostering more meaningful critique-driven refinements.
Key Results
- Across five tasks—dialog, summarization, QA, math, coding—RCO outperforms baselines with a 15% increase in average Critique Utility (CU) and 2-3 point improvements in Response Quality Score (RQS). Particularly in mathematical reasoning and code generation, the improvements are most pronounced, validating the method’s effectiveness in complex reasoning domains.
- Smaller models trained with RCO surpass larger baseline models, demonstrating parameter efficiency. Compared to DPCO, RCO shows stronger generalization across tasks and better alignment with human preferences, especially in critique relevance and response refinement.
- Human evaluations and RewardBench tests confirm that RCO-generated critiques are more helpful and lead to responses that better meet user expectations, indicating practical advantages in deployment scenarios.
Significance
This work bridges critique generation and response refinement, enabling LLMs to self-improve without extensive human supervision. By directly optimizing critique utility derived from response preferences, it addresses the bottleneck of costly human annotations and enhances model robustness. The framework paves the way for scalable, autonomous model improvement, with broad implications for AI safety, content quality, and industry deployment, especially in automated customer service, content moderation, and intelligent tutoring systems.
Technical Contribution
The paper introduces a novel training paradigm where critique utility (CU) acts as a scalar reward, derived from response preferences, to supervise critic models. It combines Bayesian regularization with a least-squares objective, enabling nuanced reward modeling. Unlike prior preference-based methods like DPO, RCO explicitly links critique quality to response refinement, fostering more effective critique-refinement loops. The approach demonstrates parameter efficiency and strong cross-task generalization, advancing the theoretical understanding of critique-driven self-improvement.
Novelty
This is the first work to utilize refinement outcome preferences directly as reward signals for critic training, bypassing the need for explicit critique quality annotations. The introduction of Critique Utility as a unified metric to guide critique generation and response refinement marks a significant innovation, establishing a new paradigm in self-supervised critique learning for LLMs.
Limitations
- The approach depends heavily on the accuracy of the preference model; biases or errors in preference judgments can impair training effectiveness, especially in ambiguous or complex tasks.
- Computational costs are high due to multiple sampling and preference evaluations, limiting scalability in real-world applications.
- The reliance on response refinement may introduce biases if the refinement process itself is flawed or overly conservative, requiring further robustness improvements.
Future Work
Future directions include integrating multi-modal data to enhance critique relevance, reducing reliance on preference models via self-supervised signals, and exploring adaptive sampling strategies to lower computational costs. Extending the framework to multilingual and multi-domain settings, as well as incorporating reinforcement learning with human feedback, will further improve robustness and applicability across diverse AI tasks.
AI Executive Summary
Large language models (LLMs) have made significant strides in natural language understanding and generation, yet their capacity for self-evaluation and critique remains limited. Traditional evaluation methods often rely on human annotations or simple scoring metrics, which are costly and do not always align with practical improvements. Critique ability—identifying flaws and guiding responses—is crucial for autonomous self-improvement, especially in complex tasks like mathematical reasoning or code generation. However, existing methods primarily train critique models for evaluation, not for directly enhancing response quality.
This paper introduces a novel framework called Refinement-oriented Critique Optimization (RCO), which fundamentally shifts how critique models are trained. Instead of relying on explicit critique preference judgments, RCO leverages a feedback loop where critiques generated by critic models guide actor models to produce refined responses. The core innovation is the Critique Utility (CU) metric, which quantifies how much a critique contributes to better responses. CU is computed by comparing the refined responses' preferences over initial responses using a preference model, effectively turning response improvements into reward signals for critic training.
The methodology involves generating multiple critiques and refinements across diverse tasks—dialogue, summarization, question answering, math, and coding—using state-of-the-art models like LLaMA and Qwen. Critic models are trained to maximize CU through a regularized least-squares objective, enabling nuanced reward modeling. Extensive experiments demonstrate that RCO-trained critics outperform traditional methods and open-source baselines, with significant gains in critique quality and response refinement scores. Human evaluations further confirm the practical benefits, showing more helpful critiques and better response improvements.
The significance of this work lies in its ability to foster scalable, autonomous self-improvement in LLMs, reducing dependence on costly human annotations. It opens new avenues for deploying AI systems that can critique, learn, and evolve independently, promising advances in AI safety, content quality, and industry applications. Future research will explore multi-modal extensions, efficiency improvements, and broader domain adaptation, pushing the boundaries of self-supervised AI learning.
Deep Dive
Abstract
Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce \textbf{R}efinement-oriented \textbf{C}ritique \textbf{O}ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method's effectiveness in enhancing LLM critique-refinement loops.