Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
AdaRFT uses adaptive curriculum learning to halve training time and boost reasoning accuracy in large language models.
Key Findings
Methodology
This work introduces AdaRFT, integrating Proximal Policy Optimization (PPO) with a dynamic difficulty adjustment mechanism. The core involves estimating problem difficulty scores, then selecting samples closest to a target difficulty T, which is continuously updated based on reward signals. The process includes: calculating the difference between each sample’s difficulty and T, selecting the B samples with minimal difference, generating responses via the policy πθ, computing average reward Ravg, and updating policy with RL algorithms. The target difficulty T is smoothed with a tanh function and clipped within bounds, ensuring training remains in an optimal difficulty zone. This lightweight extension does not modify reward functions or model architecture. Experiments on datasets like AMC, AIME, and IMO demonstrate that AdaRFT accelerates convergence, reduces training time by up to 50%, and improves final reasoning accuracy across different model scales and data distributions.
Key Results
- In experiments with Qwen 2.5 7B and 1.5B models, AdaRFT reduced training time by approximately 50%, achieving a final accuracy of 46.92% on the Math 500 benchmark, outperforming baseline PPO and static curriculum methods.
- Across skewed and uniform difficulty distributions, AdaRFT consistently demonstrated faster convergence and higher accuracy, especially in imbalanced data regimes where static sampling failed.
- Ablation studies confirmed that setting the target reward β to 0.5 (roughly 50% success rate) yields optimal training efficiency and model performance, validating the intermediate success curriculum hypothesis.
Significance
This approach addresses key bottlenecks in large-scale RL fine-tuning—namely, sample inefficiency and high computational costs—by enabling models to learn more effectively through adaptive difficulty modulation. Its simplicity, compatibility with standard RL algorithms, and significant efficiency gains make it highly valuable for deploying large models in academic research and industry applications, such as automated theorem proving, legal reasoning, and financial analysis. It paves the way for scalable, resource-efficient training pipelines that can adapt to diverse data and task complexities.
Technical Contribution
The primary innovation lies in embedding a reward-driven difficulty adjustment mechanism into RL-based fine-tuning, replacing static or staged curricula. The algorithm dynamically estimates problem difficulty, filters samples based on proximity to a target difficulty T, and updates T in a smooth, stable manner. Theoretical analysis links reward variance to the KL divergence between initial and optimal policies, providing a principled basis for setting the target reward β. This framework enhances training stability, convergence speed, and generalization, representing a significant step forward in curriculum-based RL for large language models.
Novelty
This work is the first to implement a fully adaptive, reward-based difficulty modulation mechanism in large-scale RL fine-tuning of LLMs. Unlike prior staged or static curricula, AdaRFT continuously and automatically adjusts training difficulty in response to model performance, avoiding manual stage design and repeated sampling overhead. Its simplicity and effectiveness open new avenues for efficient structured reasoning training, setting a new standard for curriculum learning in RL contexts.
Limitations
- The method relies on accurate difficulty estimation; noisy or biased scores can impair curriculum effectiveness. In highly complex tasks, difficulty metrics may need refinement.
- In extreme data distributions—either overly challenging or trivial—the adaptive mechanism may struggle to maintain optimal learning progress without further hyperparameter tuning.
- Computational costs, while reduced, remain significant for very large models or highly complex tasks, necessitating further optimization for industrial-scale deployment.
Future Work
Future research will focus on improving difficulty estimation methods, possibly integrating internal model representations for end-to-end difficulty learning. Extending the framework to multi-task and multi-modal settings could further enhance its robustness and applicability. Additionally, exploring multi-agent curriculum strategies and automated hyperparameter tuning may unlock even greater efficiency and generalization in large-scale RL training.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, yet their training, especially for structured tasks like mathematical reasoning, remains resource-intensive. Reinforcement fine-tuning (RFT) offers a promising avenue to enhance model capabilities by optimizing policies through reward signals. However, traditional RFT approaches are often hampered by inefficiencies—requiring extensive samples and computational resources, which limit scalability. To address this, the present study introduces AdaRFT, a novel algorithm that employs adaptive curriculum learning to dynamically regulate training difficulty based on the model’s recent reward feedback.
The core idea is simple yet powerful: training on tasks that are neither too easy nor too hard accelerates learning. AdaRFT maintains a target difficulty T, which is continuously adjusted using a smooth, reward-driven update rule. By selecting samples closest to T, the method ensures the model consistently trains on appropriately challenging problems. This process is integrated seamlessly with standard RL algorithms like PPO, requiring only minimal modifications.
Extensive experiments on datasets such as AMC, AIME, and IMO demonstrate that AdaRFT significantly reduces training time—up to 50%—while maintaining or improving reasoning accuracy. For instance, in the Qwen 2.5 7B model, training time was cut in half, with accuracy reaching 46.92%. Across different data distributions and model sizes, the method proved robust, especially in imbalanced scenarios where fixed curricula faltered.
The significance of this work lies in its simplicity, efficiency, and broad applicability. By enabling models to adaptively focus on the right difficulty level, AdaRFT paves the way for scalable, resource-efficient training pipelines. Its compatibility with existing RL frameworks and minimal hyperparameter tuning make it a practical tool for advancing large-scale structured reasoning tasks.
Looking forward, future directions include refining difficulty estimation, extending to multi-task and multi-modal learning, and exploring multi-agent curriculum strategies. Overall, AdaRFT represents a meaningful step toward more intelligent, efficient training of large models, with substantial implications for both academia and industry.
Deep Dive
Abstract
Reinforcement finetuning (RFT) has shown great potential for enhancing the mathematical reasoning capabilities of large language models (LLMs), but it is often sample- and compute-inefficient, requiring extensive training. In this work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a method that significantly improves the efficiency of RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the difficulty of training problems based on the model's recent reward signals, ensuring that the model consistently trains on tasks that are challenging but solvable. This adaptive sampling strategy accelerates learning by maintaining an optimal difficulty range, avoiding wasted computation on problems that are too easy or too hard. AdaRFT requires only a lightweight extension to standard RFT algorithms like Proximal Policy Optimization (PPO), without modifying the reward function or model architecture. Experiments on competition-level math datasets demonstrate that AdaRFT improves convergence efficiency and reasoning performance. Given problem-level difficulty annotations, AdaRFT reduces RFT training time by up to 2 times across data distributions and model scales, offering a more scalable and effective RFT framework.