Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
Woodpecker Distillation uses weak models to diagnose and correct local reasoning bugs, improving mathematical reasoning accuracy by 1.4% on average.
Key Findings
Methodology
The approach involves inserting short correction patches generated by a weak probe model into a strong model's reasoning prefix. By evaluating the impact of these patches on the final outcome, the method constructs a contrastive soft teacher distribution that emphasizes successful interventions. The strong model is then trained to match this distribution, internalizing the correction effect without directly copying the patches. This process leverages contrastive learning, KL regularization, and Jensen-Shannon gating to filter noise and focus on informative positions, enabling the model to learn how local interventions influence future reasoning trajectories.
Key Results
- Across five mathematical reasoning benchmarks, Woodpecker Distillation improved the average accuracy from 53.4% to 54.8%, with notable gains of 10.0 and 13.3 percentage points on AIME 2024 and AIME 2025 respectively.
- The method outperformed direct imitation and self-rejection fine-tuning baselines, demonstrating the effectiveness of contrastive local supervision, especially on high-difficulty problems.
- Ablation studies confirmed that negative interventions, JS gating, and diversity in weak probe patches are critical for performance, with each component contributing significantly to the final results.
Significance
This work addresses a fundamental challenge in large language models: correcting localized reasoning errors that cause brittle failures. By leveraging weak models as diagnostic tools, it introduces a novel paradigm for model debugging and knowledge transfer, emphasizing local correction signals over global solutions. The approach enhances the robustness and interpretability of models in complex reasoning tasks, opening avenues for safer and more reliable AI systems in education, scientific research, and industry. It also bridges the gap between weak supervision and high-performance reasoning, offering a scalable, data-efficient framework for model improvement.
Technical Contribution
The key innovation lies in contrastive distillation, which constructs a soft teacher distribution from success and failure cases of local interventions. This contrasts with traditional imitation learning that copies entire trajectories. The framework integrates KL regularization and Jensen-Shannon divergence to stabilize training and filter noise, ensuring the model internalizes the causal effect of local repairs. It also introduces a novel use of weak probe models as diagnostic tools rather than teachers, shifting the focus from full solution imitation to local correction learning, thus enabling more fine-grained and effective model refinement.
Novelty
This is the first work to utilize weak models for local reasoning bug diagnosis and contrastive distillation, emphasizing the impact of local interventions on future reasoning distributions. Unlike prior methods that rely on full trajectory imitation or global supervision, this approach leverages the differential effects of successful and unsuccessful patches, providing a new perspective on model debugging and knowledge transfer. Its contrastive, distribution-based framework offers a more nuanced and scalable way to improve reasoning accuracy.
Limitations
- The current evaluation is limited to mathematical reasoning benchmarks; applicability to other domains such as code generation or factual reasoning remains untested and requires further validation.
- The method relies on verifier feedback, which can be computationally expensive, especially at scale, potentially limiting real-world deployment.
- The effectiveness depends on the weak probe model's quality; poor weak models may generate uninformative or noisy patches, reducing the correction signal's reliability.
Future Work
Future research will explore extending the framework to multi-modal reasoning tasks, integrating visual and textual data. Improving the efficiency of verifier-based feedback and developing more robust weak models are also key directions. Additionally, combining this approach with reinforcement learning or self-supervised signals could further enhance the model's ability to self-correct in diverse, real-world scenarios.
AI Executive Summary
Despite remarkable progress, large language models still struggle with localized reasoning errors, especially in complex tasks like mathematics. Traditional training approaches, such as full trajectory imitation or global fine-tuning, often fail to address these subtle flaws, leading to brittle failures that limit practical deployment. Recognizing this challenge, the authors introduce Woodpecker Distillation, a novel framework that leverages weak models as diagnostic probes to identify and correct local reasoning bugs.
This method hinges on inserting short correction patches generated by a weak probe model into a strong model’s reasoning prefix. By evaluating the impact of these patches on the final outcome, the approach constructs a contrastive soft teacher distribution that emphasizes successful interventions. The strong model then learns to internalize this correction effect by matching the teacher distribution, without directly copying the weak patches. This contrastive learning process, combined with KL regularization and Jensen-Shannon gating, filters noise and stabilizes training.
Experimental results on five mathematical reasoning benchmarks demonstrate the effectiveness of Woodpecker Distillation. The method consistently improves accuracy, with the most notable gains on high-difficulty datasets—AIME 2024 and AIME 2025—where accuracy increases by over 10 percentage points. Ablation studies confirm the importance of negative interventions and the contrastive framework, validating the core design choices.
This work advances the understanding of local reasoning bugs and offers a scalable, data-efficient way to enhance model robustness. Its implications extend beyond mathematics, promising improvements in scientific reasoning, code generation, and factual inference. While challenges remain—such as computational costs and generalization to other domains—the framework sets a new direction for model debugging and knowledge transfer, fostering more reliable and interpretable AI systems in the future.
Deep Analysis
Background
近年来，随着Transformer架构和大规模预训练模型的发展，语言模型在自然语言理解和推理任务中取得了突破性进展。早期的Seq2Seq模型逐步演变为GPT、BERT等基于自注意力机制的模型，显著提升了文本理解能力。为了应对复杂推理，诸如Chain-of-Thought（思维链）和反事实推理等技术被提出，增强模型的推理深度和解释性。尽管如此，模型在多步推理中仍频繁出现局部错误，尤其在数学题、逻辑推理等高难度任务中表现不稳定。传统的微调和监督方法在一定程度上缓解了问题，但难以根除推理中的细节性错误。近年来，知识蒸馏、对比学习等技术被引入模型调试，试图通过局部干预改善推理性能，但缺乏系统性框架。本文的贡献在于提出一种基于弱模型局部修正的对比蒸馏方法，填补了局部推理错误修正的研究空白。
Core Problem
强模型在推理任务中经常出现局部错误，导致最终答案不正确。这些错误多源于中间推理步骤的偏差或错误，传统方法难以有效识别和修正。全局微调或完整轨迹模仿成本高，效果有限。如何利用局部干预信号，自动识别并修正推理中的微小偏差，成为提升模型鲁棒性的关键难题。解决此问题有助于模型在复杂推理任务中表现更稳定，减少错误率，提升可信度。
Innovation
本研究的创新在于：1）利用弱模型生成的短修正片段作为诊断工具，揭示推理中的可修复错误；2）提出对比学习的软教师构建策略，通过比较成功与失败的修正，学习未来推理轨迹的变化；3）结合KL正则化和JS门控，有效过滤噪声，确保训练的稳定性。这一框架区别于传统的全局模仿或监督，强调局部修正对未来推理的影响，为模型调试提供了新思路。
Methodology
- �� 在强模型中采样推理轨迹，选择中间位置插入弱模型生成的短修正片段。
- �� 评估每个修正片段对最终答案的影响，将成功与失败的修正分为两组。
- �� 构建对比教师分布，基于成功与失败组的未来推理分布差异，利用KL正则化生成软目标。
- �� 训练强模型，使其输出分布逼近该对比教师分布，从而内化修正效果。
- �� 采用JS门控调节不同位置的权重，过滤噪声，提升训练效果。
Experiments
使用Qwen3-4B-Instruct-2507作为强模型，Gemma-3-4B-IT作为弱模型，在五个数学推理基准（如AIME、Math-500）上评估。通过比较未修正、直接模仿修正和对比蒸馏三种策略，验证了方法的有效性。采用验证器自动评估答案正确性，进行消融实验验证关键组件的贡献。训练过程中调节超参数η和δ，确保模型稳定收敛。
Results
在五个基准中，Woodpecker Distillation平均提升准确率1.4个百分点，最大在AIME 2025提升13.3点，显著优于对比方法。消融实验显示，负样本对比和JS门控是性能提升的关键因素，缺一不可。该方法在高难度问题集中的表现尤为优异，验证了其在复杂推理中的潜力。
Applications
该技术适用于需要高可靠性推理的场景，如自动化数学解题、法律推理、科学研究辅助等。通过引入局部修正机制，可以显著提升模型在复杂、多步推理任务中的表现，增强其鲁棒性和解释性，为工业界提供更可靠的AI解决方案。
Limitations & Outlook
目前主要在数学推理任务验证，泛化到其他推理领域（如代码生成、事实推理）尚未充分验证。依赖验证器反馈，计算成本较高，限制了大规模应用。弱模型性能不足时，修正信号可能不稳定，未来需优化弱模型和验证机制，提升泛用性。
Plain Language Accessible to non-experts
想象你在厨房做菜，遇到一道复杂的菜谱。你用平时的经验尝试做，但有时会出错。于是，你请一个经验较少的朋友帮你看看，提出一些小建议。你根据朋友的建议调整步骤，结果可能会更好。这就像用弱模型提供的小修正，帮助强模型（厨师）改正推理中的错误。通过反复比较成功和失败的建议，厨师学会了如何自己调整，做出更好菜肴。这种方法让厨师变得更聪明、更可靠，能应对更复杂的菜谱。
ELI14 Explained like you're 14
想象你在学校做数学题，有时候你会走错路，但只要有人告诉你哪里错了，你就能改正。比如，你在算一道难题时，突然想到一个新办法，结果答案变对了。这就像用弱模型给出的小提示，帮你修正推理的错误。通过不断试错和对比成功与失败的提示，你学会了自己怎么避免犯同样的错。这样一来，你的数学水平就会变得更厉害，也更能应对各种难题。这种方法就像让你变成了一个更聪明、更有经验的学生。
Glossary
Distillation (蒸馏)
一种模型训练技术，通过提取和传递有用的知识信号，优化模型性能。技术上指将复杂模型的知识转移到更简洁模型中。
本文中指利用对比学习将局部修正信号蒸馏到强模型中。
Contrastive Learning (对比学习)
一种通过比较不同样本的差异来学习有用表示的方法，强调正负样本的对比信息。
用于构建软教师分布，强化模型对成功修正的学习。
Weak Probe Model (弱模型探针)
性能较弱的模型，用于生成局部修正片段，揭示推理中的错误区域。
作为诊断工具，提供局部修正线索。
Teacher Distribution (教师分布)
由成功与失败修正对比构建的目标分布，用于指导强模型学习修正效果。
通过KL正则化逼近，内化修正影响。
KL Regularization (KL正则化)
通过Kullback-Leibler散度限制模型输出偏离原始分布，确保学习过程的稳定性。
在构建软教师时防止偏离原始推理分布。
Open Questions Unanswered questions from this research
- 1 如何将该方法扩展到多模态推理（如图像+文本）尚未验证，未来需探索多模态信息融合的有效策略。
- 2 在大规模应用中，验证器的效率和准确性仍是瓶颈，需开发更高效的验证机制。
Applications
Immediate Applications
数学自动解题
利用Woodpecker Distillation提升数学题解的准确率，适用于教育、竞赛等场景，增强模型在复杂推理中的鲁棒性。
科学研究辅助
帮助科研模型更准确地进行多步推理，减少错误，提高科研效率。
Long-term Vision
智能推理系统
未来可发展成为具备自主修正能力的智能系统，广泛应用于法律、医疗、工程等领域，推动AI的可信赖性。
Abstract
Large language models often fail on reasoning tasks despite possessing the capability to solve them. We argue that many such failures arise from localized reasoning bugs in intermediate steps rather than from global incompetence. We show that these bugs are frequently repairable: inserting a short patch generated by a weak probe model after the same strong-model reasoning prefix can redirect the trajectory toward a correct solution. However, this corrective effect is not reliably internalized by directly fine-tuning on weak patches or repaired trajectories, suggesting that the useful signal lies not in the intervention text itself, but in how it reshapes the model's future reasoning distribution. We therefore propose Woodpecker Distillation, a weak-to-strong training framework that learns from contrastive local interventions. Our method contrasts successful and unsuccessful weak-model patches at the same prefix, constructs a corrective teacher distribution from their induced future token predictions, and distills this signal into the strong model. Experiments on mathematical reasoning benchmarks show that Woodpecker Distillation consistently improves strong-model performance and outperforms direct imitation baselines.