OVD: On-policy Verbal Distillation
OVD: trajectory matching with verbal scores reduces memory, improves Web QA and math reasoning by up to 25.7%.
Key Findings
Methodology
OVD replaces token-level probability matching with trajectory matching guided by discrete verbal scores (0-9) from teacher models. It employs on-policy sampling with a rejection mechanism based on verbal feedback, enabling efficient RL training. The core algorithm integrates PPO for policy updates and a theoretical proof ensures unbiased gradient estimates. This approach avoids large vocabulary logits, drastically reducing memory usage, and supports multi-step reasoning and interactive exploration.
Key Results
- On Web QA benchmarks, OVD improves average EM by 12.9%, and on mathematical reasoning tasks, achieves up to 25.7% gain with only single random samples. It outperforms existing RL-based methods across multiple datasets like NQ, TriviaQA, HotpotQA, and GAIA, especially on complex multi-hop and real-world questions.
- Experimental validation confirms that verbal rejection sampling yields unbiased gradients and reduces variance, leading to more stable training. The method scales efficiently with longer trajectories and larger batch sizes, demonstrating superior sample efficiency and inference performance.
- Ablation studies reveal that verbal feedback granularity and rejection thresholds significantly impact results, with finer score vocabularies improving trajectory quality assessment and overall performance.
Significance
This work addresses the critical memory bottleneck in token-level distillation, especially in RL scenarios involving long sequences. By leveraging discrete verbal scores, it enables scalable, on-policy training of large models with interactive feedback, bridging the gap between model exploration and supervision. The approach enhances reasoning capabilities, facilitates deployment in resource-constrained environments, and opens new avenues for integrating human-like feedback into model training, advancing both academic research and practical AI applications.
Technical Contribution
The paper introduces a novel trajectory matching framework that replaces large vocabulary logits with discrete verbal scores, supported by a rigorous theoretical analysis guaranteeing unbiased gradient estimates. The verbal rejection sampling mechanism dynamically balances exploration and expert supervision, enabling stable, efficient RL training. The combination of these innovations significantly reduces memory footprint, supports long-horizon reasoning, and enhances model exploration, setting a new standard for knowledge distillation in large language models.
Novelty
This is the first work to incorporate discrete verbal feedback into on-policy RL distillation, moving away from traditional token-level supervision. The integration of trajectory matching with verbal rejection sampling provides a scalable, memory-efficient alternative, enabling models to explore output spaces freely while maintaining theoretical guarantees. This approach fundamentally differs from prior methods that rely on full probability distributions, marking a significant innovation in model compression and interactive learning.
Limitations
- The discrete verbal scores may lack the granularity needed for highly nuanced reasoning tasks, potentially limiting fine-grained supervision.
- The method's reliance on accurate verbal feedback from teachers could introduce biases or errors, especially in complex or ambiguous scenarios.
- Training involves extensive sampling and interaction, which, despite efficiency gains, still incurs considerable computational costs. Future work should focus on optimizing sampling strategies and feedback mechanisms.
Future Work
Future research will explore richer feedback modalities, including continuous scores and multi-dimensional evaluations, to improve supervision quality. Integrating multi-modal feedback (visual, auditory) and self-supervised signals could further enhance model capabilities. Additionally, developing more efficient sampling algorithms and applyingOVD to broader tasks like dialogue systems and real-time reasoning will be key directions.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, yet their reasoning capabilities remain limited by computational and memory constraints. Traditional knowledge distillation methods, which rely on token-level probability matching, face severe scalability issues when applied to long sequences and reinforcement learning scenarios. These methods demand enormous memory to store logits over vast vocabularies, making training impractical for complex tasks requiring multi-step reasoning.
In response, this paper introduces OVD (On-policy Verbal Distillation), a novel framework that replaces token-level supervision with trajectory-level matching guided by discrete verbal scores (0-9). This shift significantly reduces memory consumption, enabling scalable, on-policy reinforcement learning. The core idea involves using teacher models to evaluate reasoning trajectories with verbal feedback, which are then sampled via a rejection mechanism that filters out low-quality trajectories. This process ensures that the student model learns from high-quality, on-policy experiences, balancing exploration and supervision.
The technical foundation of OVD combines policy gradient methods like PPO with a theoretical guarantee of unbiased gradient estimation. The verbal rejection sampling mechanism dynamically adjusts the mixture of student-generated and teacher demonstrations, facilitating stable training and efficient exploration. Extensive experiments on Web question answering and mathematical reasoning datasets demonstrate that OVD outperforms existing methods, with improvements up to 12.9% in EM scores and 25.7% in math benchmarks, even when trained with minimal samples.
This approach offers a scalable, memory-efficient alternative to traditional distillation, fostering models capable of complex, multi-step reasoning in resource-constrained environments. Its success paves the way for future integration of richer, multi-modal feedback and broader application in interactive AI systems, marking a significant step forward in model compression and reasoning capabilities.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂的目标是生产高质量的产品。传统的方法就像每个工人都要逐个检查每个零件，确保每个都完美，但这样太慢也太费力。现在，工厂引入了一个新系统，工人们只用给每个产品打个0到9的评分，快速判断产品的整体质量。这个评分系统可以帮助工人更快地筛选出好产品，也能告诉他们哪里需要改进。通过不断调整评分标准，工厂逐渐提高了生产效率和产品质量。这就像论文中的Verbal评分机制，用简单的数字替代复杂的概率分布，让工厂（模型）更快、更聪明地学习，生产出更好的产品（答案）。
ELI14 Explained like you're 14
想象你在学校里参加一个比赛，老师给你每个答案打分，从0到9，告诉你答得好不好。你可以根据老师的评分，知道自己哪里做得不错，哪里还需要努力。以前，老师会告诉你每个字的详细评价，比如每个字的正确率，但这样很麻烦，也占用很多空间。现在，老师只用一个数字，既快又方便。你可以用这个数字来调整自己的答题策略，慢慢变得更厉害。这个方法就像论文里的Verbal评分，用简单的数字帮助你快速学习，避免繁琐的细节，让你更聪明、更快地进步。
Abstract
Knowledge distillation offers a promising path to transfer reasoning capabilities from large teacher models to efficient student models; however, existing token-level on-policy distillation methods require token-level alignment between the student and teacher models, which restricts the student model's exploration ability, prevent effective use of interactive environment feedback, and suffer from severe memory bottlenecks in reinforcement learning. We introduce On-policy Verbal Distillation (OVD), a memory-efficient framework that replaces token-level probability matching with trajectory matching using discrete verbal scores (0--9) from teacher models. OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback, and avoids token-level alignment, allowing the student model to freely explore the output space. Extensive experiments on Web question answering and mathematical reasoning tasks show that OVD substantially outperforms existing methods, delivering up to +12.9% absolute improvement in average EM on Web Q&A tasks and a up to +25.7% gain on math benchmarks (when trained with only one random samples), while also exhibiting superior training efficiency. Our project page is available at https://OVD.github.io