Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher
Proposes Teach-to-Reason (T2R), a framework using self-improving teachers and contrastive supervision to enhance reasoning in CXR VQA, outperforming baselines.
Key Findings
Methodology
T2R integrates a self-competitive teacher with a contrastive scoring mechanism, iteratively strengthening the teacher to generate superior reasoning references. The Reasoner is trained against these references, with a case-wise reward design that preserves original task rewards when informative, and recovers supervision during reward collapse. The approach employs multi-stage training, progressively improving teacher models and leveraging pairwise comparisons via LLM judges to guide fine-grained optimization. Extensive experiments on six chest X-ray VQA datasets demonstrate significant performance gains over RLVR, Frozen Teacher, and Judge Reward baselines, validating the effectiveness of contrastive supervision and iterative teacher enhancement.
Key Results
- Across six datasets, T2R consistently achieves the highest or near-highest accuracy at both 2B and 4B scales, with improvements of 5-10% over baselines, especially excelling in SLAKE and MIMIC-CXR-VQA.
- Iterative teacher improvement results in denser supervision signals, with the ratio of non-zero advantages remaining high throughout training, leading to better reasoning chain quality.
- The case-wise reward design effectively recovers supervision in degenerate groups, maintaining model performance even when task rewards become uninformative.
Significance
This work advances the state-of-the-art in medical multimodal reasoning by addressing the limitations of coarse reward signals. It introduces a novel training paradigm combining self-competition and contrastive supervision, significantly improving the reliability and interpretability of VQA models in clinical settings. The approach offers a scalable solution for continual self-improvement, with broad implications for deploying trustworthy AI in healthcare, especially for complex diagnostic tasks requiring detailed reasoning and justification.
Technical Contribution
Key innovations include the integration of a self-improving teacher via self-competition, a contrastive pairwise scoring mechanism for fine-grained supervision, and a case-wise reward strategy that maintains training stability during reward collapse. The multi-stage training pipeline effectively balances original task rewards with dynamic supervision signals, enabling continual model enhancement. These contributions differentiate T2R from existing methods that rely solely on coarse rewards or fixed judges, providing a flexible, scalable framework for reasoning optimization.
Novelty
This is the first work to combine iterative self-competition teachers with contrastive pairwise supervision specifically for medical VQA. Unlike prior approaches that depend on predefined rules or static judges, T2R dynamically refines reasoning references through model self-evolution, effectively addressing reward sparsity and collapse issues. Its case-wise reward design and degenerate-case handling constitute novel mechanisms that ensure persistent supervision, setting a new direction for self-supervised reasoning in multimodal AI.
Limitations
- The method relies heavily on large pretrained models and LLM-based judges, which entail high computational costs and limit accessibility in resource-constrained environments.
- Performance may degrade on out-of-distribution or highly complex cases where the judge's scoring ability is limited, requiring further robustness improvements.
- The current framework focuses on chest X-ray VQA; extending to other modalities and tasks may require additional adaptation and validation.
Future Work
Future research will explore integrating causal inference and multi-modal data fusion to enhance interpretability and robustness. Developing lightweight, efficient versions of T2R for real-time clinical deployment and extending the framework to broader medical tasks such as diagnosis and treatment planning are promising directions. Additionally, incorporating explainability modules and domain-specific priors could further improve clinical trustworthiness.
AI Executive Summary
In the rapidly evolving field of medical AI, chest X-ray visual question answering (VQA) stands out as a key challenge, demanding models not only to produce accurate answers but also to justify their reasoning processes. Traditional reinforcement learning approaches, which rely heavily on answer-level rewards, often fall short in guiding models toward high-quality, interpretable reasoning chains. These rewards are coarse and sparse, especially when group advantages diminish, leading to stagnation in learning. Addressing this critical gap, the present work introduces Teach-to-Reason (T2R), a novel framework that leverages a self-improving teacher and a contrastive supervision mechanism to enhance reasoning capabilities.
T2R’s core idea involves an iterative teacher model that undergoes self-competition, progressively generating stronger reasoning references. These references serve as dynamic targets for training the Reasoner, which is the main VQA model. Unlike fixed-rule judges, T2R employs a pairwise comparison approach, where a large language model (LLM) acts as an evaluator, comparing two reasoning chains and assigning a score based on reasoning correctness and consistency. This contrastive scoring provides a fine-grained supervision signal, guiding the Reasoner to improve its chain-of-thought (CoT) quality.
A key innovation in T2R is its case-wise reward design. When the original task reward is informative, it preserves the positive/negative partition of samples, refining within-group rankings via the contrastive scores. When the reward signal collapses, the framework reconstructs supervision by leveraging the comparison scores, effectively recovering the training signal. This adaptive mechanism ensures continuous learning even under challenging conditions.
Extensive experiments on six chest X-ray VQA datasets demonstrate that T2R outperforms baseline methods such as RLVR, Frozen Teacher, and Judge Reward, with improvements of 5-10% in accuracy. The iterative teacher enhancement process results in denser supervision signals, maintaining a high ratio of non-degenerate groups during training. The ablation studies confirm the importance of the self-competition and case-wise reward strategies, highlighting their roles in boosting reasoning performance.
Overall, T2R offers a significant step forward in medical multimodal reasoning, providing a robust, scalable, and interpretable training paradigm. Its ability to sustain supervision during reward collapse and continuously improve reasoning quality holds promise for deploying trustworthy AI systems in clinical practice. Future directions include integrating causal reasoning, expanding to other modalities, and optimizing computational efficiency, aiming to realize fully autonomous, explainable medical AI solutions.
Deep Analysis
Background
近年来，深度学习在医学影像分析中取得了巨大突破，胸部X光问答（CXR VQA）作为多模态理解的重要任务，逐渐成为评估模型推理能力的关键指标。早期工作如VQA-RAD、VQA-Med等，主要关注答案的准确性，缺乏对推理过程的解释。随着大规模预训练模型（如Med-BERT、BiomedCLIP）的出现，性能有所提升，但仍面临推理链质量不足、监督信号稀疏的问题。强化学习（RL）和判别器（Judge）等技术被引入，但多依赖粗粒度奖励，难以持续引导推理能力的优化。如何设计细粒度、持续有效的监督机制，成为当前研究的瓶颈。
Core Problem
现有方法在优化推理链质量方面效果有限，主要由于奖励信号过于粗糙，难以引导模型进行细粒度改进。在奖励信号退化或群体优势消失时，模型难以获得有效的训练信号，导致推理能力停滞甚至退化。此外，缺乏一种能持续自我提升的训练机制，限制了模型在复杂临床场景中的应用潜力。
Innovation
本研究提出T2R框架，核心创新包括：1）引入自我竞争教师，通过模型内部竞争不断增强推理参考链；2）采用对比评分机制，作为细粒度的监督信号，有效缓解奖励退化问题；3）设计案例级奖励策略，确保在奖励信号退化时依然能提供有效监督。这些创新结合，显著提升了推理链的质量和模型的诊断可信度。
Methodology
- �� 设计教师模型（Teacher）与推理器（Reasoner），教师通过自我竞争不断增强，生成更优推理参考链。
- �� 利用LLM作为判别器，对两个推理链进行对比评分，评估其推理正确性和一致性。
- �� 在训练中结合任务奖励和对比分数，设计案例级奖励，确保在奖励信号退化时依然能提供有效监督。
- �� 采用多阶段训练流程，逐步提升教师模型，优化推理器性能。
- �� 在退化场景下，利用对比分数重建训练信号，保持推理能力的持续提升。
Experiments
在六个公开胸部X光VQA数据集（如MIMIC-CXR、SLAKE）上，使用2B和4B规模的Qwen3-VL-Instruct模型进行训练。比较基线包括RLVR、Frozen Teacher和Judge Reward。训练中采用K=10参考推理链，超参数如α=0.9，γ=0.3。评估指标为模型答题准确率，通过多次判别确保结果稳健。还进行了消融实验，验证教师迭代、退化处理和对比机制的贡献。
Results
T2R在六个数据集上均优于对比方法，平均提升5-10个百分点，特别是在SLAKE和MIMIC-CXR-VQA上表现突出。逐步增强的教师模型带来持续性能提升，训练过程中竞争分数密度保持高水平，验证了自我竞争机制的有效性。退化处理机制显著提高了在奖励退化情况下的恢复能力，确保训练的稳定性和模型推理质量。
Applications
该方法适用于临床医学中的智能问答系统，能提升模型在复杂诊断场景中的推理能力。未来可结合电子病历和多模态影像信息，开发更具解释性和鲁棒性的医疗AI工具，辅助医生进行诊断决策，推动个性化医疗发展。
Limitations & Outlook
模型训练依赖大量计算资源，成本较高。判别器性能受限于LLM能力，可能在极端复杂场景下表现不足。模型在偏离训练分布的样本上仍存在性能下降的风险，未来需增强泛化能力和推理解释性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多工人（模型），他们需要完成一项复杂的任务，比如组装一台新机器。每个工人都可以提出自己的方案，但要确保方案合理、正确。工厂里有一个主管（教师模型），他会不断观察工人的方案，并通过内部比赛（自我竞争）挑选出最好的方案，然后告诉其他工人（推理器）去学习。工人们之间还会互相比较，谁的方案更合理，就会得到奖励。这样，工厂里的工人们不断学习、改进，最终能组装出高质量的机器。这个过程就像论文中的自我竞争教师和对比监督机制，通过不断优化推理链，提升模型的推理能力和解释性。
ELI14 Explained like you're 14
想象你在学校里参加一个比赛，老师会给你一些题目，让你写出解决方案。开始时，你的方案可能还不够好，但老师会让你和其他同学的方案比一比，看看谁的更合理。每次比完后，老师会告诉你哪个方案更棒，然后你会学习这个更好的方案，下一次做得更好。老师还会自己不断练习，变得越来越厉害，能给你提供更好的参考。通过这种不断比拼和学习的方式，你的解题能力会逐步提高。这就像论文里的自我竞争教师和对比评分机制，让模型不断学习、变得更聪明，最终能给出更合理的答案和推理。
Glossary
Chain-of-Thought (推理链)
指模型在推理过程中逐步展开的思考路径，用于提升推理的透明性和准确性。技术上是由一系列逻辑步骤组成的推理序列。
在论文中，强调通过优化推理链质量，提升模型的推理能力和解释性。
对比学习 (Contrastive Learning)
一种通过比较样本差异，学习判别特征的方法，增强模型区分不同类别或质量的能力。技术上涉及样本对的相似性或差异性度量。
论文中用以评估两个推理链的优劣，作为细粒度监督信号。
自我竞争 (Self-competition)
模型内部通过相互比较、竞争，推动自身性能提升的机制。常用于模型自我优化和持续学习。
教师模型通过自我竞争不断增强，生成更优推理参考。
案例级奖励 (Case-wise Reward)
针对每个样本单独设计的奖励机制，确保在奖励信号退化时还能提供有效监督。
论文中用以在奖励退化时恢复训练信号，保证推理链优化的连续性。
退化场景 (Degenerate Case)
指奖励信号或优势指标失去区分能力，导致模型难以优化的情况。
论文提出机制处理退化场景，保持训练的稳定性。
Open Questions Unanswered questions from this research
- 1 如何在更大规模、多模态、多任务环境中保持对比监督机制的稳定性和有效性？
- 2 模型在极端退化或偏离训练分布的样本上的泛化能力如何进一步提升？
- 3 结合因果推理和强化学习，能否设计更具解释性和稳健性的推理优化策略？
Applications
Immediate Applications
临床辅助诊断系统
利用T2R提升胸部X光问答模型的推理质量，为医生提供更可靠的诊断建议和解释，改善医疗决策的透明度。
医学教育与培训
通过优化推理链，帮助医学生理解复杂病例的诊断流程，提升医学教育的互动性和效果。
Long-term Vision
智能医疗AI平台
将T2R技术集成到全面的医疗AI平台，实现多模态、多任务的自主学习和持续优化，推动个性化医疗和远程诊疗。
Abstract
Chest X-ray visual question answering (CXR VQA) requires models not only to predict correct answers, but also to produce reliable medical reasoning. However, existing reinforcement-learning-based training typically relies on answer-level rewards, which are often too coarse to improve chain-of-thought (CoT) quality and can become ineffective when group-level advantages collapse to zero. We propose \textbf{Teach-to-Reason (T2R)}, a framework that introduces comparison-based supervision into CoT optimization through a self-improving \emph{Teacher} and a competition-guided \emph{Reasoner}. As the Teacher is iteratively strengthened via self-competition, the Reasoner is optimized against progressively stronger Teacher-generated references. We further introduce a case-wise reward design that preserves the original reward-induced positive/negative partition when it is informative, and restores supervision from competition scores when the original reward signal degenerates. Experiments on multiple CXR open-ended VQA benchmarks show that T2R consistently outperforms strong baselines, indicating that comparison-based supervision, when integrated in a controlled and principled manner, provides a more effective training signal for reasoning optimization.