PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners
PAINT integrates adaptive partial solution masking and sparse energy interpolation, boosting self-distilled reasoning performance.
Key Findings
Methodology
PAINT employs a context-aware re-scoring framework, dynamically adjusting the exposure of verified solutions based on rollout-reference overlap (α). It applies sparse energy-space interpolation at high-entropy token positions to calibrate teacher targets, balancing correctness and diversity. The approach involves:• Overlap-based adaptive masking controlling solution information exposure;• Entropy-ratio guided sparse energy interpolation at key positions;• Fixing teacher parameters while optimizing student parameters within a re-scoring paradigm. This mechanism enhances reasoning by controlling information flow and calibration, leading to improved performance.
Key Results
- On Qwen3-8B, PAINT improves Avg@12 by 2.1 points over the prior self-distillation baseline (from 64.8 to 66.9), outperforming GRPO (64.0).
- Across 4B and 1.7B scales, gains range from 0.8 to 1.1 points, demonstrating scalability and robustness.
- The combination of adaptive masking and sparse interpolation effectively reduces overfitting to single reference traces, increasing reasoning diversity and accuracy.
Significance
This work addresses the fundamental challenge of balancing supervision informativeness and trajectory diversity in large language models’ reasoning training. By integrating context-aware re-scoring with adaptive information control, PAINT advances the state-of-the-art in reasoning tasks, especially in scenarios with verified solutions but limited external supervision. Its techniques enable models to generalize better, handle complex problems more accurately, and reduce reliance on stronger teachers, thus opening new avenues for AI applications in scientific and educational domains.
Technical Contribution
The paper introduces a novel combination of context-induced Gibbs re-scoring, adaptive partial solution masking based on overlap scores, and sparse energy interpolation guided by entropy ratios. These innovations allow dynamic control over the amount and location of solution information exposed during training, improving the model’s reasoning capabilities without increasing training complexity. The theoretical guarantees on monotonicity and optimal placement of masks provide a solid foundation for future adaptive training strategies.
Novelty
This is the first work to unify context-dependent re-scoring with adaptive partial masking and sparse energy-space calibration in self-distillation for reasoning tasks. Unlike prior methods that rely on fixed or full solution exposure, PAINT dynamically adjusts the information flow based on alignment metrics, preventing overfitting and promoting reasoning diversity. Its integration of these techniques represents a significant step forward in training models capable of complex, multi-step reasoning.
Limitations
- The method depends on the accuracy of overlap scores as proxies for alignment, which may falter in highly ambiguous or noisy scenarios.
- Sparse energy interpolation, while effective, may not fully address errors in highly complex or adversarial reasoning tasks.
- Hyperparameter sensitivity and increased computational overhead pose challenges for large-scale deployment.
Future Work
Future research could explore multi-modal reasoning extensions, integrating visual or symbolic data. Adaptive curriculum strategies, including dynamic masking and interpolation parameters, could further improve robustness. Additionally, applying these techniques to other reasoning domains like theorem proving or scientific discovery, and reducing computational costs, remain promising directions.
AI Executive Summary
In recent years, large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and reasoning. However, their performance on complex mathematical and scientific tasks remains limited by training paradigms that often lack fine-grained supervision aligned with inference states. Traditional supervised fine-tuning (SFT) provides dense but static targets, while reinforcement learning with sparse rewards (RLVR) guides exploration but suffers from high variance. Self-distillation approaches, especially privileged on-policy self-distillation, have emerged as promising middle ground, leveraging verified solutions as training context without requiring external teachers.
This paper introduces PAINT, a novel training framework that enhances reasoning by adaptively controlling the exposure of verified solutions during self-distillation. The core idea is to dynamically mask parts of the solution based on rollout-reference overlap, encouraging broader reasoning when the model is less aligned, and sharpening guidance as alignment improves. Additionally, PAINT employs sparse energy-space interpolation at high-entropy token positions, gently calibrating teacher targets to prevent overfitting to a single reference trajectory.
The mechanism hinges on a theoretical reinterpretation of re-scoring as a Gibbs distribution induced by context-dependent energy landscapes. By controlling the amount of privileged information revealed and calibrating targets locally, PAINT effectively balances correctness and diversity. Empirical results on Qwen3-8B, 4B, and 1.7B models show consistent improvements over prior self-distillation baselines, with average gains of 2.1 points on Avg@12 scores across math benchmarks. The approach outperforms or matches state-of-the-art methods like GRPO, while requiring fewer rollout tokens and less computational overhead.
Overall, PAINT offers a significant step forward in training large models for complex reasoning tasks. Its adaptive masking and sparse interpolation strategies provide a flexible, theoretically grounded framework for balancing supervision informativeness and model generalization. Future work may extend this paradigm to multi-modal reasoning, theorem proving, and scientific discovery, further broadening the impact of this innovative approach.
Deep Analysis
Background
推理能力是衡量大规模语言模型性能的关键指标。早期方法主要依赖监督微调（SFT）和蒸馏技术，提供密集的目标，但在推理任务中存在信息暴露不足的问题。近年来，强化学习奖励（RLVR）尝试引导模型探索，但奖励稀疏且方差大，效果有限。自蒸馏（OPSD）利用验证解作为训练信号，改善了这一局限，但仍面临信息暴露与推理多样性之间的矛盾。随着模型规模的扩大，如何在保证推理准确性同时，增强模型的推理多样性，成为研究热点。
Core Problem
核心问题在于如何在推理训练中合理调节验证解信息的暴露程度。过度暴露会导致模型过拟合单一轨迹，影响泛化；而信息不足则限制模型推理能力。现有方法难以动态适应不同推理阶段的需求，缺乏有效的调节机制，限制了模型在复杂推理任务中的表现。如何在保证推理准确性的同时，提升模型的推理多样性和鲁棒性，是亟待解决的关键难题。
Innovation
本文提出PAINT，结合上下文重评分机制与稀疏能量插值，创新性地解决推理中的信息调节问题。具体包括：• 基于rollout-reference重叠度的动态遮掩策略，调节验证解信息暴露；• 利用entropy比值在关键位置进行能量空间插值，微调教师目标；• 固定教师模型参数，优化学生模型推理能力。这些创新实现了推理信息的自适应调节，有效提升模型性能，避免过拟合单一轨迹。
Methodology
- �� 利用rollout-reference重叠度（α）指标动态调节验证解的遮掩比例，低重叠度暴露更多信息，高重叠度隐藏部分内容；• 在高entropy位置应用稀疏能量插值，微调教师目标，减少偏差；• 固定教师参数，训练学生模型，结合上下文重评分机制，强化推理能力；• 采用KL散度进行目标校准，确保目标的密集性与多样性平衡；• 通过调节遮掩比例与插值强度，实现推理信息的自适应调控。
Experiments
实验在Qwen3-8B、4B、1.7B模型上进行，数据来自OpenThoughts数学推理子集，包含3个竞赛基准（AIME 2024/2025，HMMT 2025）。采用rollout预算为每题1个1024-token的单次轨迹，训练过程中固定教师模型参数。对比SFT、GRPO、OPSD等方法，评估Avg@12得分提升，验证PAINT在推理准确率和泛化能力上的优势。还进行了消融实验，验证遮掩策略和稀疏插值的贡献。
Results
在Qwen3-8B模型上，PAINT在Avg@12指标上平均提升2.1分（从64.8提升至66.9），优于GRPO（64.0）。在4B和1.7B模型上也实现了0.8至1.1分的提升。实验显示，遮掩比例与插值策略的调节显著改善模型推理表现，验证了方法的有效性。稀疏能量插值减少了模型对单一参考的依赖，增强了推理多样性。
Applications
该方法适用于数学、科学推理、自动证明等领域，特别在有限标注或验证解可用的场景。通过调节验证信息的暴露程度，提升模型在复杂推理任务中的准确性和鲁棒性。未来可结合多模态信息，拓展到更广泛的推理场景，推动AI在科研、教育等行业的深度应用。
Limitations & Outlook
当前方法依赖于参考轨迹的重叠度指标，可能在极端偏离参考的情况下表现不足。稀疏插值在复杂推理场景中可能受限，调参复杂且对超参数敏感。模型训练成本较高，需进一步优化算法效率和泛化能力。
Plain Language Accessible to non-experts
想象你在做一道数学题，就像在厨房里做菜。你有一个完整的食谱（验证解），但每次做菜时，你可以选择只看部分步骤（遮掩部分信息），这样可以训练自己更灵活地应对不同的情况。为了确保菜做得好，你会在关键步骤用少量调料（能量插值）微调味道，避免只依赖某一种做法。这样一来，你既能学会多样的做法，又不容易被单一食谱绑死。PAINT的方法也是这样：它让模型在学习推理时，既看到完整的答案，又学会在不同信息量下调整自己的推理策略，最终变得更聪明、更灵活。
ELI14 Explained like you're 14
想象你在玩一个解谜游戏，你可以看到全部线索，也可以只看部分线索。为了变得更聪明，你会尝试只看一些重要的线索，然后用自己的方法推理出答案。这样做可以让你学会在信息不全的情况下也能找到答案，不会只依赖全部线索。PAINT就像这样，它教模型在推理时，有时候只看部分验证解，然后用微调的方法，让模型学会在不同信息量下都能推理得很好。这样，模型就变得更聪明、更灵活，能应对各种复杂的问题。
Abstract
Improving large language model (LLM) reasoning requires supervision that is both aligned with the model's own test-time states and informative at the token level. Reinforcement learning with verifiable rewards provides on-policy exploration but offers sparse, high-variance credit; supervised fine-tuning and distillation provide dense targets but often train on fixed trajectories or rely on stronger teachers. Recent privileged on-policy self-distillation explores a middle ground by scoring student rollouts with the same model under verified solution context. We revisit this setting through a contextual re-scoring lens: for reasoning, the important choices are not only whether privileged context is available, but how much of it should be revealed and where its distribution should shape the student. We propose PAINT (Partial-solution Adaptive INterpolated Training), which masks the verified solution according to rollout-reference overlap and applies a small energy-space interpolation on a sparse set of entropy-mismatch token positions. Across competition-level math benchmarks, PAINT consistently improves over a strong prior on-policy self-distillation baseline at all three Qwen3 scales. On Qwen3-8B, it raises macro Avg@12 by 2.1 points over this prior baseline and 2.9 points over GRPO.