Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning
Proposed a difficulty-aware SFT-then-RL framework to enhance small language model reasoning.
Key Findings
Methodology
The paper presents a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets to enhance small language model reasoning. A Bridge mechanism is introduced in the SFT stage to convert complex reasoning traces into more learnable supervision signals. In the RL stage, Critique Fine-Tuning is applied to unsolved hard samples, turning failures into diagnostic, repair, and new reasoning trace supervision.
Key Results
- Experiments on five reasoning benchmarks show that this method consistently outperforms representative SFT, distillation, and RL baselines on two small language models.
- On GSM8K-Platinum, the Qwen2.5-0.5B-Instruct model's accuracy improved to 53.10%.
- In the Llama3.2-1B-Instruct model, the pass@8 metric increased to 84.29% after using the Bridge mechanism.
Significance
This research highlights the importance of coordinating data difficulty across SFT and RL stages, demonstrating that a well-designed data strategy can significantly enhance small language model reasoning. It holds significant academic value and offers new insights for industrial applications in resource-constrained environments.
Technical Contribution
The technical contribution lies in proposing a new data allocation strategy that, combined with the Bridge mechanism and Critique Fine-Tuning, transforms complex reasoning samples into learnable supervision signals, significantly improving small language model reasoning.
Novelty
This is the first to introduce a difficulty-aware data allocation strategy in the SFT-then-RL framework, addressing the challenge of handling complex reasoning samples that previous methods struggled with.
Limitations
- The method relies on teacher-generated supervision signals, which may introduce additional computational costs and biases.
- Its effectiveness on non-mathematical domains and larger language models remains to be verified.
Future Work
Future research can explore the application of this framework to larger language models and non-mathematical domains, and develop more efficient automatic scoring methods to reduce computational costs.
AI Executive Summary
Small language models are gaining attention for their efficiency and cost advantages in resource-constrained applications. However, existing methods to enhance reasoning capabilities often lack targeted data selection, making it challenging for models to effectively learn complex reasoning skills. This paper proposes a difficulty-aware SFT-then-RL framework that introduces a Bridge mechanism in the SFT stage to convert complex reasoning traces into more learnable supervision signals, and applies Critique Fine-Tuning in the RL stage to turn failures into diagnostic, repair, and new reasoning trace supervision.
Experimental results show that this method consistently outperforms representative SFT, distillation, and RL baselines across five reasoning benchmarks, with significant performance improvements on GSM8K-Platinum and Llama3.2-1B-Instruct models. This research holds significant academic value and offers new insights for industrial applications in resource-constrained environments.
While the method shows excellent performance in mathematical and logical reasoning, its effectiveness on non-mathematical domains and larger language models remains to be verified. Future research can explore the application of this framework to broader domains and develop more efficient automatic scoring methods to reduce computational costs.
Deep Analysis
Background
In recent years, the development of large language models (LLMs) has significantly enhanced reasoning capabilities. However, small language models (SLMs) with fewer parameters have relatively weaker reasoning abilities. Existing research mainly focuses on improving SLM reasoning through distillation and post-training, but lacks targeted data selection.
Core Problem
The core problem is how to allocate data of different difficulties across the SFT and RL stages to maximize small language model reasoning capabilities. Existing methods often struggle with complex reasoning samples, making it difficult for models to learn effectively.
Innovation
The core innovation of this paper is the introduction of a difficulty-aware SFT-then-RL framework that transforms complex reasoning samples into learnable supervision signals through the Bridge mechanism and Critique Fine-Tuning. The Bridge mechanism adjusts the complexity of reasoning traces to make them more suitable for SLM learning.
Methodology
- �� Data Allocation: Training data is divided into simple, medium, and hard categories.
- �� Bridge Mechanism: In the SFT stage, hard samples are converted into more learnable supervision signals.
- �� Critique Fine-Tuning: In the RL stage, failure samples are turned into diagnostic, repair, and new reasoning trace supervision.
Experiments
Experiments were conducted on five benchmarks including GSM8K-Platinum and MAWPS, using two models: Qwen2.5-0.5B-Instruct and Llama3.2-1B-Instruct. The main evaluation metrics were accuracy and pass@8.
Results
The experimental results show that the difficulty-aware framework improves performance on all benchmarks for both models, with significant accuracy improvements on GSM8K-Platinum.
Applications
This method is suitable for scenarios where small language models need to be deployed in resource-constrained environments, such as smart assistants on mobile devices and real-time translation applications.
Limitations & Outlook
The method relies on teacher-generated supervision signals, which may introduce additional computational costs and biases. Additionally, its effectiveness on non-mathematical domains and larger language models remains to be verified.
Plain Language Accessible to non-experts
Imagine you're learning a new skill, like riding a bike. Initially, you need someone to hold you up, similar to the SFT stage, helping you grasp the basics. Once you can balance, you start riding on your own, akin to the RL stage, consolidating the skills you've mastered. If you fall, don't worry, it's part of the learning process. Our method is like a smart coach, knowing when to support you, when to let you try on your own, and telling you what went wrong when you fall, helping you do better next time.
ELI14 Explained like you're 14
Imagine playing a super tough video game. At first, you need some hints and help, like our SFT stage, to understand the game rules. Once you're familiar, you can tackle harder levels on your own, like the RL stage, reinforcing the skills you've learned. If you fail, no worries, our method will tell you what went wrong, just like game hints, helping you pass next time. Isn't that cool?
Glossary
SFT (Supervised Fine-Tuning)
A technique to fine-tune models using labeled data to learn new skills.
Used in this paper to enhance small language model reasoning capabilities.
RL (Reinforcement Learning)
A technique to optimize model decisions using reward signals, primarily for consolidating mastered skills.
Used in this paper to consolidate reasoning skills already accessible to the model.
Bridge Mechanism
A method to convert complex reasoning traces into more learnable supervision signals.
Used in the SFT stage to handle hard samples.
Critique Fine-Tuning
A method to turn RL stage failure samples into diagnostic, repair, and new reasoning trace supervision.
Used to enhance learning in the RL stage.
pass@8
A metric to evaluate model performance in reasoning tasks, indicating the probability of success in at least one out of eight attempts.
Used to evaluate the effectiveness of the SFT stage.
Open Questions Unanswered questions from this research
- 1 How to apply this framework to non-mathematical domains? Current methods perform well in mathematical reasoning but are unclear in other domains.
- 2 How to reduce the computational cost of teacher-generated supervision signals? Current methods may introduce additional computational burdens.
Applications
Immediate Applications
Smart Assistants on Mobile Devices
By enhancing small language model reasoning capabilities, smart assistants can operate more efficiently on resource-constrained mobile devices.
Long-term Vision
Real-Time Translation Applications
Using small language models in real-time translation applications to improve translation accuracy and speed.
Abstract
Post-training Small Language Models (SLMs) for reasoning typically follows an SFT-then-RL pipeline, yet existing work rarely considers what data should be learned at each stage. We argue that data strategy should be aligned with the distinct roles of SFT and RL: SFT is better suited for acquiring not-yet-mastered reasoning skills, while RL is better suited for consolidating skills that the model can already partially access. Based on this principle, we propose a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets. For hard samples in the SFT stage, we introduce a Bridge mechanism that transforms raw teacher-generated reasoning traces into more learnable supervision for SLMs. For hard samples that remain unsolved during RL, we apply Critique Fine-Tuning by converting all-zero-reward failures into diagnostic, repair, and new reasoning trace supervision for the next SFT stage. Experiments on two SLMs across five reasoning benchmarks show that our method consistently improves over representative SFT, distillation, and RL baselines. Our results highlight the importance of coordinating data difficulty across SFT and RL for effective SLM reasoning post-training.