Rubric-based On-policy Distillation
Proposes ROPD, a rubric-based on-policy distillation method, achieving 10× sample efficiency.
Key Findings
Methodology
ROPD employs teacher-student contrast to automatically generate prompt-specific rubrics, which are then used by a verifier to score student responses as rewards for policy optimization. The framework consists of a Rubricator (extracts task-specific criteria) and a Verifier (evaluates responses), eliminating the need for teacher logits. Multiple teachers and contrastive comparisons ensure robustness and generalization. This semantic scoring approach replaces dense probability distributions, enabling efficient knowledge transfer in black-box settings.
Key Results
- In tasks like AIME24/25, HMMT, and GPQA, ROPD outperforms logit-based OPD, with up to 10× sample efficiency gains. For example, on HMMT25 (Nov.), performance increased from 7.08 to 41.67, a +34.6 point absolute improvement. It also surpasses existing black-box distillation methods, demonstrating strong adaptability.
- In white-box scenarios, ROPD achieves performance comparable or superior to logit-based methods, with over 20% gains. The results highlight that high-level semantic rewards better capture correctness than token-level likelihoods, especially in complex reasoning tasks.
- Ablation studies confirm multi-teacher contrast, shared rubrics, and blind scoring are critical for performance. These components help avoid path dependency and bias, ensuring stable convergence and high-quality distillation across diverse tasks.
Significance
This work addresses the limitations of traditional logit-dependent distillation, enabling effective model alignment in proprietary and API-based environments. By leveraging structured semantic rubrics, it significantly improves sample efficiency and robustness in complex reasoning tasks. The approach broadens the applicability of model distillation, offering a flexible, interpretable, and scalable solution for industry and academia, fostering progress in model compression, knowledge transfer, and AI interpretability.
Technical Contribution
The paper introduces a novel rubric-based OPD framework that replaces dense logit signals with structured semantic criteria, facilitating black-box distillation. The design includes a multi-teacher contrast mechanism, shared rubrics for stability, and a blind verifier to reduce bias. This approach enhances interpretability, robustness, and transferability across architectures, setting a new benchmark in sample efficiency and model alignment. The theoretical and empirical results demonstrate its superiority over existing methods.
Novelty
This is the first work to formalize a rubric-based, semantic scoring approach for on-policy distillation, especially in black-box settings. Unlike previous methods relying on probability distributions, ROPD uses interpretable criteria to guide learning, enabling effective knowledge transfer without internal model access. Its multi-teacher contrast and shared rubric strategies represent innovative steps toward scalable, robust distillation.
Limitations
- The quality of generated rubrics depends on the contrastive comparison; poorly designed or insufficient criteria may limit effectiveness, especially in highly complex tasks.
- Automating rubric generation for diverse, multi-modal, or highly abstract tasks remains challenging and requires further research.
- The approach's performance in extremely low-resource or noisy environments needs further validation, and computational overhead from multiple teachers and contrastive comparisons could be a concern.
Future Work
Future directions include automating and optimizing rubric generation via meta-learning or large pre-trained models, extending the framework to multi-modal tasks, and exploring adaptive contrast strategies. Additionally, integrating self-supervised signals and reinforcement learning could further improve robustness and generalization. The goal is to develop fully autonomous, scalable, and interpretable distillation systems suitable for real-world deployment across diverse AI applications.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized natural language processing, but effectively transferring knowledge from powerful teachers to smaller students remains challenging. Traditional distillation methods rely heavily on teacher logits, which require internal access to model probabilities—a limitation in proprietary or API-restricted environments. To address this, researchers have explored response-level signals, but these often lack the granularity needed for complex reasoning.
This paper introduces ROPD, a novel rubric-based on-policy distillation framework that overcomes these barriers. Instead of relying on dense probability distributions, ROPD automatically generates task-specific scoring standards—called rubrics—by contrasting teacher and student responses. These rubrics serve as high-level semantic guides, which are then evaluated by a verifier to produce reward signals. This process enables effective model alignment in black-box settings, where only textual responses are accessible.
The core innovation lies in using structured, interpretable criteria to guide learning, which significantly improves sample efficiency—up to ten times better than traditional methods. Extensive experiments across multiple benchmarks, including math and reasoning tasks, demonstrate that ROPD not only outperforms existing black-box distillation techniques but also remains competitive in white-box scenarios without access to teacher logits. The approach’s robustness is validated through ablation studies, confirming the importance of multi-teacher contrast, shared rubrics, and blind scoring.
Beyond empirical success, ROPD offers a new perspective on model distillation—shifting from probability matching to goal-oriented, semantic guidance. Its ability to operate efficiently and interpretably across diverse architectures and environments paves the way for scalable, low-cost AI deployment. Future work will focus on automating rubric generation, extending to multi-modal tasks, and integrating reinforcement learning to further enhance model robustness and generalization, promising a transformative impact on AI model compression and knowledge transfer.
Deep Dive
Abstract
On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.