On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
CHORD framework harmonizes SFT and RL via dynamic weighting, improving LLMs' performance.
Key Findings
Methodology
CHORD framework integrates Supervised Fine-Tuning (SFT) as a dynamically weighted auxiliary objective within the Reinforcement Learning (RL) process. It employs a global coefficient μ and a token-wise weighting function ϕ(·) to control the influence of off-policy expert data, promoting on-policy exploration while mitigating disruption from off-policy data.
Key Results
- CHORD outperformed baseline models in math reasoning tasks, with a 2.4% improvement on AMC and a 1.0% increase on AIME24.
- In tool-use tasks, CHORD-ϕ achieved superior overall performance, with a BFCL score of 78.5.
- By controlling token-wise weights, CHORD-ϕ effectively avoided overfitting, enhancing model generalization.
Significance
The CHORD framework effectively combines off-policy expert data with on-policy exploration, addressing the instability issues present in traditional SFT-then-RL methods. It provides a new perspective for training large language models in academia and offers more efficient solutions for complex tasks in industry.
Technical Contribution
CHORD introduces a dynamic weighting mechanism, integrating SFT as part of the RL process rather than a separate stage. With a global coefficient and token-wise weighting function, CHORD achieves fine-grained control over the influence of off-policy data, offering new engineering possibilities.
Novelty
CHORD is the first to integrate SFT as a dynamically weighted auxiliary objective within RL, significantly improving training stability and performance through its innovative dual-control mechanism.
Limitations
- CHORD may still experience performance fluctuations when handling expert data significantly different from the model's established patterns.
- Adjusting the decay strategy for μ across different tasks increases tuning complexity.
Future Work
Future research could explore CHORD's applicability to more tasks and optimize its dynamic weighting strategy to accommodate different types of expert data.
AI Executive Summary
In training large language models (LLMs), Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two common post-training paradigms. However, traditional SFT-then-RL methods often face performance instability. To address this challenge, the paper proposes the CHORD framework, which integrates SFT as a dynamically weighted auxiliary objective within the RL process.
The CHORD framework employs a global coefficient μ and a token-wise weighting function ϕ(·) to achieve fine-grained control over the influence of off-policy expert data. Experimental results show that CHORD excels in both mathematical reasoning and tool-use tasks, significantly outperforming baseline models.
CHORD's innovation lies in its dual-control mechanism, effectively combining off-policy expert data with on-policy exploration, addressing the performance instability issues in traditional methods. Future research can further optimize its dynamic weighting strategy to suit different types of tasks and data.
Deep Analysis
Background
In recent years, large language models (LLMs) have made significant progress in the field of natural language processing. Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two key post-training techniques that enhance LLM performance. SFT relies on high-quality expert data for pattern imitation, while RL encourages exploration through direct feedback on policy generation.
Core Problem
Although SFT and RL each have their strengths, the traditional SFT-then-RL method often faces performance instability when combining the two. Expert data may significantly differ from the model's established patterns, leading to performance fluctuations and overfitting during training.
Innovation
The CHORD framework integrates SFT as a dynamically weighted auxiliary objective within the RL process. It employs a global coefficient μ and a token-wise weighting function ϕ(·) to achieve fine-grained control over the influence of off-policy expert data, promoting on-policy exploration while mitigating disruption from off-policy data.
Methodology
- �� Introduce a global coefficient μ to dynamically adjust the weighting of SFT and RL loss functions.
- �� Design a token-wise weighting function ϕ(·) to control the learning signal strength of each token.
- �� Achieve a smooth transition from off-policy imitation to on-policy optimization through dynamic weighting.
Experiments
Experiments were conducted on mathematical reasoning and tool-use tasks using the OpenR1-Math-220k and ToolAce datasets. Baselines included the original model, SFT-only, RL-only, and SFT+RL. Key metrics included AMC, AIME24, and BFCL.
Results
CHORD outperformed baseline models in math reasoning tasks, with a 2.4% improvement on AMC and a 1.0% increase on AIME24. In tool-use tasks, CHORD-ϕ achieved superior overall performance, with a BFCL score of 78.5.
Applications
CHORD can be applied to complex tasks requiring the integration of expert knowledge and policy exploration, such as autonomous driving and intelligent customer service. Its dynamic weighting mechanism offers greater flexibility in handling diverse data.
Limitations & Outlook
CHORD may still experience performance fluctuations when handling expert data significantly different from the model's established patterns. Adjusting the decay strategy for μ across different tasks increases tuning complexity.
Plain Language Accessible to non-experts
Imagine a chef in a kitchen. Traditional SFT-then-RL methods are like learning a recipe first and then trying various spices to improve the taste. However, this approach may lead to unstable flavors. CHORD is like a smart assistant that can dynamically adjust the spice proportions based on ingredient changes, ensuring each dish achieves the best flavor.
ELI14 Explained like you're 14
Imagine you're playing a game with two tasks: one is to mimic a pro player's moves, and the other is to explore new skills. The traditional method is to learn to mimic first, then explore. CHORD is like a smart coach that helps you mimic and explore simultaneously, dynamically adjusting your training plan based on your performance, making you stronger faster!
Glossary
Supervised Fine-Tuning (SFT)
Fine-tuning a model using high-quality expert data to mimic expert response patterns.
Used as the source of off-policy data in CHORD.
Reinforcement Learning (RL)
Optimizing a model's policy through exploration based on direct feedback from policy generation.
Core mechanism for on-policy exploration in CHORD.
Dynamic Weighting
Adjusting the weights of loss functions dynamically based on training needs.
Used in CHORD to balance the influence of SFT and RL.
Global Coefficient μ
A global parameter controlling the weighting of SFT and RL loss functions.
Used in CHORD to dynamically adjust the influence of off-policy data.
Token-wise Weighting Function ϕ(·)
Adjusts the learning signal strength of each token based on its generation probability.
Enhances training stability in CHORD.
Open Questions Unanswered questions from this research
- 1 How to optimize CHORD's dynamic weighting strategy for more tasks?
- 2 How does CHORD perform with extreme off-policy data?
Applications
Immediate Applications
Autonomous Driving
Enhances decision-making in autonomous systems by integrating expert driving data with real-time feedback.
Long-term Vision
Intelligent Customer Service
Combines expert knowledge and real-time user feedback through dynamic weighting, offering superior service quality.
Abstract
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs). Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data. To address this, we present a novel investigation into the unified view of SFT and RL through an off-policy versus on-policy lens. We propose CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting, which reframes SFT not as a separate stage but as a dynamically weighted auxiliary objective within the on-policy RL process. Based on an analysis of off-policy expert data's influence at both holistic and granular levels, we incorporate a dual-control mechanism in CHORD. Specifically, the framework first employs a global coefficient to holistically guide the transition from off-policy imitation to on-policy exploration, and then applies a token-wise weighting function that enables granular learning from the expert, which promotes on-policy exploration and mitigates disruption from off-policy data. We conduct extensive experiments across various practical tasks, providing empirical evidence that CHORD achieves a stable and efficient learning process. By effectively harmonizing off-policy expert data with on-policy exploration, CHORD demonstrates significant improvements over baselines. We release the implementation at https://github.com/modelscope/Trinity-RFT/tree/main/examples/mix_chord to inspire further research.