Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
Proposes an expand-then-compress framework using multi-teacher policy distillation, improving Qwen3-1.7B's reasoning by 2.0%-8.3%.
Key Findings
Methodology
The study introduces an expand-then-compress framework combining Residual Group Relative Policy Optimization (RGRPO) and Reliability-Gated Teacher-Union On-policy Distillation (TU-OPD). In the expansion stage, RGRPO trains a sequence of teachers from a common initialization to cover examples not yet covered by the teacher union. In the compression stage, TU-OPD allows the student to learn from its own response prefixes, with only reliable teachers contributing, weighted by per-example quality.
Key Results
- The Qwen3-1.7B student model outperforms the strongest individual teacher by 2.0%, 8.3%, and 6.9% in mathematical reasoning, code generation, and instruction following, respectively.
- Experiments show that the multi-teacher strategy in the expansion stage significantly enhances the overall performance of the student model, especially in mathematical reasoning where the student model's average performance exceeds any single teacher.
- Ablation studies reveal that the complementarity among teachers is effectively integrated during the compression stage, enhancing the student's comprehensive capabilities.
Significance
This study addresses the issue of incomplete coverage by single reinforcement learning teachers in reasoning tasks through multi-teacher policy distillation, significantly improving student model performance across multiple domains. The framework provides a new approach for academia and industry to leverage diverse reasoning modes, enhancing model generalization.
Technical Contribution
The proposed expand-then-compress framework fundamentally differs from existing methods by constructing and compressing multi-teacher strategies. By combining RGRPO and TU-OPD, the method not only expands the coverage of reasoning solutions but also preserves specialized teacher behavior during compression, offering new engineering possibilities.
Novelty
This is the first study to combine multi-teacher strategies with reasoning solution diversity. Compared to previous methods, this study not only focuses on teacher selection but also achieves comprehensive utilization of teacher capabilities through the expand-then-compress framework.
Limitations
- The method may still face challenges in handling extremely complex reasoning tasks, especially when there is significant variability among teachers.
- In some cases, the reliability assessment of teachers may not be precise enough, affecting the final performance of the student model.
Future Work
Future work could explore applying this framework to larger datasets and more complex tasks to further enhance reasoning capabilities. Additionally, researching how to automate teacher selection and combination is an important direction.
AI Executive Summary
In complex reasoning tasks, a single reinforcement learning teacher often covers only a limited set of solution modes. To address this issue, this paper proposes an expand-then-compress framework that enhances student model reasoning capabilities through multi-teacher policy distillation.
The framework consists of two stages: expansion and compression. In the expansion stage, Residual Group Relative Policy Optimization (RGRPO) is used to train a series of teachers to cover examples not yet covered by the teacher union. In the compression stage, Reliability-Gated Teacher-Union On-policy Distillation (TU-OPD) compresses the teachers' capabilities into a single student model.
Experimental results show that the Qwen3-1.7B student model outperforms the strongest individual teacher by 2.0%, 8.3%, and 6.9% in mathematical reasoning, code generation, and instruction following, respectively. This framework provides a new approach for academia and industry to leverage diverse reasoning modes, enhancing model generalization.
Deep Analysis
Background
In recent years, reinforcement learning has made significant progress in reasoning tasks. However, a single reinforcement learning teacher often covers only a limited set of solution modes, leading to suboptimal performance in certain reasoning tasks. To address this issue, researchers have begun exploring multi-teacher policy distillation methods to enhance model generalization.
Core Problem
The issue of incomplete coverage by single reinforcement learning teachers in reasoning tasks is a significant challenge. Due to the complexity of reasoning tasks, correct solutions are rarely unique, and a single teacher may only capture a subset of solution modes, leading to poor performance in uncovered areas.
Innovation
The proposed expand-then-compress framework combines Residual Group Relative Policy Optimization (RGRPO) and Reliability-Gated Teacher-Union On-policy Distillation (TU-OPD). In the expansion stage, RGRPO trains a series of teachers to cover examples not yet covered by the teacher union. In the compression stage, TU-OPD compresses the teachers' capabilities into a single student model.
Methodology
- �� Expansion stage: Use RGRPO to train a series of teachers to cover examples not yet covered by the teacher union.
- �� Compression stage: Use TU-OPD to compress the teachers' capabilities into a single student model.
- �� Reliability assessment: Weight teachers' contributions based on per-example quality.
Experiments
Experiments were conducted in three domains: mathematical reasoning, code generation, and instruction following, using the Skywork-OR1-RL-Data and IFBench datasets. Evaluation metrics included Avg.@8 and Pass@8. The experimental design included four Residual-GRPO teachers and a student model.
Results
The Qwen3-1.7B student model outperforms the strongest individual teacher by 2.0%, 8.3%, and 6.9% in mathematical reasoning, code generation, and instruction following, respectively. Experiments show that the multi-teacher strategy in the expansion stage significantly enhances the overall performance of the student model.
Applications
The framework can be used to enhance model performance in complex reasoning tasks, applicable to fields such as mathematical reasoning, code generation, and instruction following. Its multi-teacher policy distillation method leverages diverse reasoning modes to enhance model generalization.
Limitations & Outlook
Despite its strong performance across multiple domains, the framework may still face challenges in handling extremely complex reasoning tasks. Additionally, the reliability assessment of teachers may not be precise enough, affecting the final performance of the student model.
Plain Language Accessible to non-experts
Imagine a school where each teacher excels in a different subject. Some are great at math, others at science. Our goal is to train a student who can learn the best from all these teachers. We first let each teacher focus on their specialty, then integrate their knowledge into one student. This way, the student excels in all subjects, not just one.
ELI14 Explained like you're 14
Imagine you're playing a game with four different characters, each with their own special skills. You want to create a super character that has all these skills. First, you make each character stronger in their field, then you combine their skills into one character. Now, you've got an unbeatable character that can defeat all the enemies in the game!
Glossary
Residual Group Relative Policy Optimization (RGRPO)
A method for training multiple teacher models to cover examples not yet covered by existing teachers.
Used in the expansion stage to train teacher models.
Teacher-Union On-policy Distillation (TU-OPD)
A method for compressing the capabilities of multiple teacher models into a single student model.
Used in the compression stage to integrate teacher capabilities.
Qwen3-1.7B
A student model enhanced through multi-teacher policy distillation to improve reasoning capabilities.
Used in experiments to evaluate the framework's performance.
Skywork-OR1-RL-Data
A dataset used for training and evaluating models in mathematical reasoning and code generation.
One of the datasets used in the experiments.
IFBench
A benchmark dataset for evaluating instruction-following tasks.
One of the datasets used in the experiments.
Open Questions Unanswered questions from this research
- 1 How can this framework be applied to larger datasets to enhance reasoning capabilities?
- 2 How can teacher selection and combination be automated to optimize model performance?
Applications
Immediate Applications
Mathematical Reasoning
The framework can enhance model performance in mathematical reasoning tasks, applicable in education and research.
Long-term Vision
General Artificial Intelligence
Through multi-teacher policy distillation, it may lead to more powerful general AI capable of excelling in various tasks.
Abstract
A single reinforcement-learning run can produce a strong reasoner yet an incomplete teacher: it often amplifies only a subset of the valid solution modes. We argue that reinforcement learning (RL)-trained policies should therefore be viewed as local probes of a multi-basin reasoning solution manifold, rather than as globally reliable supervisors. Based on this view, we propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation. In the expansion stage, Residual Group Relative Policy Optimization (RGRPO) trains a sequence of teachers from a common initialization and redirects each later round toward examples not yet covered by the accumulated teacher union. In the compression stage, reliability-gated Teacher-Union On-policy Distillation (TU-OPD) lets the student learn from its own response prefixes. For each example, only reliable teachers contribute, and their sampled-token OPD losses are weighted by their per-example quality. We further introduce Consensus-Residual Decomposition, which preserves a winner teacher's excess token preferences over its reliable peers, preventing specialist behavior from being suppressed during teacher aggregation. Experiments on mathematical reasoning, code generation, and instruction following show that the resulting Qwen3-1.7B student consistently outperforms the strongest individual teacher across all three domains, yielding relative improvements of 2.0%, 8.3%, and 6.9%, respectively, while retaining single-model inference. These results establish a simple but powerful principle: stronger students can be obtained not by selecting a single better teacher, but by deliberately constructing and compressing a complementary teacher union.