DeepCritic: Deliberate Critique with Large Language Models
DeepCritic introduces a two-stage framework leveraging Qwen2.5-72B-Instruct to enhance mathematical critique, outperforming GPT-4o and DeepSeek with significant accuracy gains.
Key Findings
Methodology
This work employs a two-phase training approach: first, generating 4.5K long-form critiques using Qwen2.5-72B-Instruct for supervised fine-tuning, cultivating step-wise critique capabilities. Second, applying reinforcement learning with human-labeled or Monte Carlo-automated correctness data to deepen critique depth. The model surpasses GPT-4o and DeepSeek on error detection benchmarks, providing more detailed feedback for correction. The process involves iterative critique generation, multi-perspective evaluation, and meta-critiquing, enabling the model to reflect and improve its judgments.
Key Results
- DeepCritic-7B-RL-PRM800K achieves F1 scores of 77.3, 60.1, and 74.0 on GSM8K, PRM800K, and ProcessBench respectively, outperforming GPT-4o (58.2) and DeepSeek (63.4) by over 10%.
- The model demonstrates higher sensitivity in error detection, especially in multi-step reasoning, with reduced false positives and more precise feedback.
- Reinforcement learning significantly enhances the model’s critical reflection, validated through ablation studies showing improved depth and multi-angle verification capabilities.
Significance
This research advances the field of automated supervision by enabling large models to perform deliberate, multi-perspective critique of complex reasoning steps. It addresses the bottleneck of shallow feedback in current models, paving the way for scalable, self-improving AI systems in education, scientific research, and AI safety. The integration of automatic data generation and deep reflection mechanisms offers a new paradigm for model self-assessment, reducing reliance on costly human annotations and fostering continuous improvement.
Technical Contribution
The paper introduces a novel two-stage training pipeline combining supervised fine-tuning with reinforcement learning, emphasizing multi-perspective critique and meta-evaluation. It innovatively merges initial and in-depth critiques into a deliberate long-form critique, enabling iterative evaluation and self-correction. The automatic data generation via Monte Carlo sampling reduces annotation costs and enhances robustness, setting a new standard for self-supervised critique models in complex reasoning tasks.
Novelty
This is the first comprehensive framework integrating long-text critique generation with reinforcement learning to improve step-wise reasoning critique in large language models. Unlike prior work limited to shallow feedback or single-pass fine-tuning, this approach emphasizes deliberate, multi-angle reflection and self-correction, significantly elevating critique depth and accuracy in mathematical reasoning.
Limitations
- Despite improvements, the model still struggles with highly ambiguous or extremely complex problems, partly due to noise in automatically generated data.
- Training costs remain high, especially during reinforcement learning, limiting scalability for very large models or broader domains.
- The current focus on mathematical reasoning needs to be extended to other reasoning types for broader applicability.
Future Work
Future directions include expanding the critique framework to multimodal reasoning, integrating visual or symbolic data, and reducing training costs through more efficient algorithms. Additionally, incorporating human feedback in a hybrid supervision scheme could further improve critique depth and robustness, enabling more generalizable self-supervised AI systems.
AI Executive Summary
The rapid evolution of large language models (LLMs) has unlocked unprecedented capabilities in natural language understanding and reasoning. However, ensuring the correctness and reliability of their outputs remains a critical challenge, especially in complex domains like mathematics. Traditional supervision methods rely heavily on human annotations, which are costly and not scalable. To address this, recent research has explored using LLMs themselves as critique agents, capable of evaluating and guiding their own outputs.
This paper introduces DeepCritic, a novel two-stage framework designed to significantly enhance the critique ability of LLMs in mathematical reasoning tasks. The first stage involves generating a large dataset of 4.5K long-form critiques using Qwen2.5-72B-Instruct, which includes multi-perspective evaluations and meta-critiquing of each reasoning step. These critiques serve as supervised data for fine-tuning a critique model, enabling it to produce deliberate, in-depth evaluations.
In the second stage, reinforcement learning is employed to further deepen the critique capability. This is done either with human-labeled data or automatically generated correctness estimates via Monte Carlo sampling. The resulting model, DeepCritic-7B-RL-PRM800K, demonstrates remarkable improvements over existing models like GPT-4o and DeepSeek, achieving F1 scores exceeding 77% on multiple benchmarks.
The significance of this work lies in its potential to transform AI supervision, making it more scalable, accurate, and self-sufficient. By enabling models to critically analyze and reflect on their reasoning, it opens pathways toward autonomous self-improvement in complex reasoning domains. Despite its success, challenges remain in extending the approach to broader tasks and reducing computational costs. Future research will focus on multimodal critique, efficiency, and integrating human feedback to further advance this promising direction.
Deep Dive
Abstract
As Large Language Models (LLMs) are rapidly evolving, providing accurate feedback and scalable oversight on their outputs becomes an urgent and critical problem. Leveraging LLMs as critique models to achieve automated supervision is a promising solution. In this work, we focus on studying and enhancing the math critique ability of LLMs. Current LLM critics provide critiques that are too shallow and superficial on each step, leading to low judgment accuracy and struggling to offer sufficient feedback for the LLM generator to correct mistakes. To tackle this issue, we propose a novel and effective two-stage framework to develop LLM critics that are capable of deliberately critiquing on each reasoning step of math solutions. In the first stage, we utilize Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for supervised fine-tuning. Each seed critique consists of deliberate step-wise critiques that includes multi-perspective verifications as well as in-depth critiques of initial critiques for each reasoning step. Then, we perform reinforcement learning on the fine-tuned model with either existing human-labeled data from PRM800K or our automatically annotated data obtained via Monte Carlo sampling-based correctness estimation, to further incentivize its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct not only significantly outperforms existing LLM critics (including the same-sized DeepSeek-R1-distill models and GPT-4o) on various error identification benchmarks, but also more effectively helps the LLM generator refine erroneous steps through more detailed feedback.