Critique-Guided Distillation for Robust Reasoning via Refinement
Proposes Critique-Guided Distillation (CGD), leveraging teacher critiques to improve reasoning, achieving +7% average gains on benchmarks.
Key Findings
Methodology
CGD framework treats teacher critiques as supervision signals during training, guiding the student to refine answers based on error feedback. It involves three steps: student generates initial answer, teacher critiques it, then produces a refined answer. The student learns to map (prompt, initial answer, critique) to the refined answer by minimizing negative log-likelihood. During inference, only prompts are needed, with the model directly producing refined answers, avoiding format drift and additional inference steps. Extensive experiments across five model families show consistent improvements, especially in mathematical reasoning tasks, with up to +15% gains on AMC23 and +12.2% on MATH-500.
Key Results
- CGD improves performance significantly: +15% on AMC23, +12.2% on MATH-500, and better results on AIME24/25, indicating enhanced reasoning quality. It maintains instruction-following abilities, with no degradation (−21.3% in CFT). Cross-family tests show strong transferability and resource efficiency, with minimal training overhead.
- Across models like LLaMA and S1.1, CGD outperforms SFT, distillation, and CFT baselines, especially on math and reasoning benchmarks. Ablation studies confirm the importance of critique as a training signal, with performance drops up to 114% when removed. The method demonstrates robustness and broad applicability, including out-of-distribution tasks.
- Results indicate that training on teacher critiques enables models to internalize error correction, leading to longer reasoning chains and higher accuracy. The approach effectively balances reasoning depth and inference efficiency, making it suitable for real-world applications requiring reliable reasoning.
Significance
This work advances the field by addressing the limitations of traditional fine-tuning, which often results in shallow imitation and poor generalization. By integrating teacher critiques into training, CGD significantly enhances models’ reasoning robustness and transferability without increasing inference complexity. It offers a practical, resource-efficient paradigm for developing reasoning-centric AI systems, with potential impacts across education, scientific research, and automated decision-making. The approach also opens avenues for further research into error-aware learning and self-correction mechanisms, promising to elevate AI reasoning capabilities to new levels.
Technical Contribution
CGD introduces a novel training paradigm that decouples critique generation from inference, using teacher critiques solely as supervision signals during training. This avoids format drift and capability degradation common in previous critique-based methods. The method leverages explicit, error-specific feedback to guide models in internalizing correction strategies, resulting in improved reasoning accuracy and robustness. It demonstrates high transferability across model architectures and tasks, with minimal additional computational cost, thus offering a scalable solution for reasoning enhancement.
Novelty
This is the first work to utilize teacher critiques exclusively as a training supervision signal, rather than generating critiques during inference. Unlike prior methods like CFT, which suffer from format drift and capability loss, CGD trains models to consume critiques internally, enabling direct answer generation at inference. Its simplicity, efficiency, and broad applicability mark a significant innovation in reasoning model training, setting a new standard for error-aware learning.
Limitations
- The effectiveness depends on the quality of teacher critiques; poor critiques can mislead the model. Ensuring high-quality, relevant feedback remains a challenge.
- In highly complex, multi-step reasoning tasks, critiques may not fully capture the intricacies, limiting the model's ability to internalize complex logical structures.
- Training with critiques increases complexity and computational cost, especially for large datasets. Further work is needed to optimize critique generation and utilization.
Future Work
Future research could focus on automating high-quality critique generation, possibly via reinforcement learning or unsupervised methods. Exploring multi-modal critiques, extending to visual reasoning, and integrating with other self-correction techniques could further enhance model capabilities. Additionally, applying CGD to real-world tasks like scientific discovery, legal reasoning, and complex decision-making holds promising potential for broad societal impact.
AI Executive Summary
In recent years, large language models (LLMs) have demonstrated remarkable capabilities across diverse NLP tasks. However, their reasoning abilities often remain superficial, limited by traditional fine-tuning methods that teach models what to output without internalizing the reasoning process. Existing approaches like Chain-of-Thought prompting and critique-based fine-tuning have made progress but face issues such as format drift, inference latency, and capability degradation. To address these challenges, Kapusuzoglu et al. propose Critique-Guided Distillation (CGD), a novel training paradigm that leverages teacher critiques as supervisory signals during training. Unlike prior methods that require models to generate critiques or perform multi-pass inference, CGD trains models to consume teacher-provided critiques and produce refined answers directly, without additional inference overhead.
The core innovation of CGD lies in decoupling critique consumption from critique generation. During training, the student model learns to interpret explicit error feedback from teachers, internalizing error-aware reasoning strategies. This process involves three key steps: generating an initial answer, receiving a critique, and producing a refined answer conditioned on both. The training objective minimizes the negative log-likelihood of the teacher’s refined answer, guiding the model to internalize correction mechanisms. At inference, the model only needs the prompt, enabling single-pass, efficient reasoning.
Extensive experiments across five model families and multiple benchmarks demonstrate CGD’s effectiveness. The method achieves an average 7% improvement over standard distillation, with gains up to 15% on AMC23 and 12.2% on MATH-500. It outperforms critique fine-tuning (CFT) and standard distillation, especially on challenging mathematical and reasoning tasks, while preserving instruction-following capabilities. Notably, CGD transfers well across models and out-of-distribution tasks, including code generation, despite being trained on non-code data.
These results position CGD as a practical, resource-efficient approach for enhancing reasoning in large models. Its ability to internalize error correction without inference-time overhead makes it highly suitable for deployment in real-world applications such as education, scientific research, and automated reasoning systems. The study opens new avenues for research into error-aware learning, self-correction, and scalable reasoning enhancement, promising to push the boundaries of AI reasoning capabilities in the coming years.
Deep Analysis
Background
近年来，随着大规模预训练模型的兴起，微调技术成为模型适应特定任务的主要手段。早期工作如GPT、BERT等通过微调实现了广泛应用，但在复杂推理任务中表现仍有限。链式推理（Chain-of-Thought, CoT）和示例引导（Few-shot prompting）技术提升了推理能力，但存在泛化不足和推理延迟问题。Critique Fine-Tuning（CFT）等方法引入批评机制，改善了错误检测与修正，但在训练中出现格式漂移和能力退化。当前研究旨在通过训练模型理解和利用教师批评，提升推理的鲁棒性和泛化能力，解决传统方法的瓶颈。
Core Problem
核心问题在于：如何在不增加推理复杂度的情况下，显著提升模型推理能力。传统微调仅让模型模仿答案，缺乏对错误的理解和修正能力。多轮推理虽能改善表现，但推理过程复杂且计算成本高。CFT等方法在训练中引入批评，但推理时需要多轮交互，影响效率。如何让模型在训练中学习利用批评，推理时能直接输出正确答案，成为亟待解决的难题。
Innovation
本研究提出了Critique-Guided Distillation（CGD），核心创新在于：
1) 在训练阶段利用教师批评作为监督信号，指导模型学习错误检测与修正；
2) 在推理阶段，模型无需生成批评，直接输出修正答案，避免格式漂移和推理延迟；
3) 在多个模型和任务中验证，展现出优异的迁移性和资源效率。这一策略突破了多轮推理和批评生成的限制，为推理能力提升提供了新路径。
Methodology
- �� 输入：任务提示和学生模型的初始回答。• 生成：学生模型生成初步答案。• 批评：教师模型基于学生回答生成批评，指出错误。• 修正：教师模型在批评基础上生成修正答案。• 训练：模型学习映射（提示、学生回答、批评）→修正答案，最小化负对数似然。• 推理：仅输入提示，模型直接输出修正答案，无需生成批评或多轮交互。
Experiments
采用WebInstruct和MetaMathQA两个数据集，训练五个模型家族（如LLaMA、Qwen）。对比SFT、Distilled SFT和CFT等基线，评估数学推理、竞赛题和指令遵循能力。设置统一训练轮次和超参数，确保公平性。重点在于：模型在数学任务中的提升、泛化能力和推理质量。
Results
CGD在AMC23和MATH-500中分别提升15%和12.2%，在AIME24/25中表现优异，Pass@1显著提高。保持指令遵循能力，避免CFT的性能下降（−21.3%）。跨模型验证显示，CGD在不同模型家族中均有优势，迁移性强。消融实验确认批评作为训练信号的关键作用，性能大幅下降。整体结果表明，CGD能有效提升推理能力且训练资源合理。
Applications
该方法适用于需要高推理能力的应用场景，如教育辅导、科研辅助、自动编程等。只需在训练阶段引入批评信息，无需推理时额外步骤，便于集成到现有系统中。未来可结合自动批评生成和多模态信息，拓展到视觉推理和跨领域任务，推动AI智能水平的全面提升。
Limitations & Outlook
依赖高质量批评，批评质量低会影响训练效果。复杂推理任务中，批评表达和理解仍有限。训练成本较高，尤其在大规模数据集上。未来需优化批评生成机制，提升批评质量，降低训练成本，增强模型在极端复杂任务中的表现。
Plain Language Accessible to non-experts
想象你在厨房里做菜。每次你做菜时，厨师（教师模型）会品尝并指出哪里做得不好，然后告诉你怎么改进。你学会了根据厨师的建议调整自己的做法，下次就能做得更好。这个过程就像模型学习批评指导：在训练时，模型听到别人的批评，学会自己修正错误，但在真正用的时候，它可以自己直接做出更好的答案，就像你不用再问厨师，自己就能做出美味的菜。这种方法让模型变得更聪明、更可靠，就像厨艺水平提升一样。
ELI14 Explained like you're 14
想象你在学校写作业。有时候你会写错答案，老师（模型）会批评你的答案，告诉你哪里错了，然后你根据老师的建议改正。慢慢地，你学会了自己发现错误，自己改正答案。等你长大了，写作业时不用老师批评也能写得很好，因为你已经学会了自己检查和改正。这就像模型学习批评指导的方法：在训练中，模型听老师的批评，学会修正自己，但在用的时候，它可以自己直接给出正确答案，不需要再问老师。这样，模型变得更聪明、更有用，就像你变成了写作高手一样。
Glossary
Critique (批评)
在模型训练中，教师模型提供的关于学生回答错误或不足的详细反馈，帮助模型学习改正。技术上为文本描述，内容指向错误分析。
用在训练阶段，指导模型理解错误并修正答案。
Distillation (蒸馏)
一种模型压缩技术，将大型教师模型的知识迁移到较小的学生模型中，保持性能的同时减小模型规模。技术上通过模仿输出或中间表示实现。
本文中用于将教师模型的推理能力传递给学生模型。
Chain-of-Thought (推理链)
引导模型逐步推导答案的技术，通过生成中间推理步骤增强理解能力。技术上为多轮生成的推理过程。
提升模型在数学和逻辑推理任务中的表现。
Fine-Tuning (微调)
在预训练模型基础上，利用特定任务数据进行的模型参数调整，以适应新任务。技术上为梯度下降优化。
本文中的模型微调策略。
Pass@k
衡量模型在k次采样中至少一次正确的概率，反映模型在少样本推理中的表现。技术上为指标。
用于评估模型在竞赛题中的推理能力。
Open Questions Unanswered questions from this research
- 1 如何自动生成高质量、多样化的教师批评以进一步提升模型性能，仍是未解难题。批评的表达需兼顾准确性和多样性，才能最大化训练效果。
- 2 在极端复杂、多步骤推理任务中，模型对批评的理解和利用能力仍有限，未来需探索更有效的批评表达和理解机制。
Applications
Immediate Applications
教育智能辅导
利用CGD训练的模型可以在数学、科学等学科中提供高质量的答题指导，帮助学生自主学习，提升推理和解题能力。
科研辅助
在科研中，模型可用于自动推理、验证复杂逻辑，减少人类工作量，提高科研效率。
Long-term Vision
AI自主推理系统
未来模型能自主学习和修正推理策略，成为具备深度理解和推理能力的智能助手，推动AI在教育、科研、决策等领域的深度应用。
Abstract
Supervised fine-tuning with expert demonstrations often produces models that imitate outputs without internalizing the reasoning processes needed for robust generalization. While critique-based approaches show promise, training models to generate critiques directly, such as Critique Fine-Tuning (CFT), can lead to output-format drift and degradation of general capabilities. We propose Critique-Guided Distillation (CGD), a training framework that decouples critique consumption from critique generation. During fine-tuning, the student is trained to refine flawed responses conditioned on teacher critiques. CGD treats critiques as a \textit{training-time-only} supervision signal, encouraging internalization of error-aware reasoning: critiques guide learning but are absent at inference. Controlled ablations confirm that these reasoning gains are directly driven by the specificity and relevance of the teacher's feedback. Across five model families, CGD consistently outperforms CFT and standard distillation on mathematical reasoning benchmarks, yielding 7\% average improvements and gains of up to +15.0\% on AMC23 and +12.2\% on MATH-500. On challenging competition problems such as AIME24 and AIME25, CGD achieves substantially higher Pass@1 and stronger performance at low Pass@k, indicating improved reasoning quality per sample. Importantly, CGD preserves general instruction-following capabilities where CFT degrades significantly ($-$21.3\% on IFEval). These results position CGD as a practical and compute-efficient intermediate training paradigm for reasoning-centric tasks without introducing architectural inference-time overhead.