On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

The development of large language models (LLMs) has revolutionized NLP, with pretraining on massive datasets followed by fine-tuning to specific tasks. SFT fine-tunes models on labeled data, improving task performance, while RL aligns models with human preferences or rules, enhancing safety and usability. Despite widespread empirical success, the theoretical relationship between these methods remains unclear. Prior work has shown that combining SFT and RL can lead to performance trade-offs, but lacks formal guarantees. Recent advances in optimization theory, spectral analysis, and non-convex landscapes provide tools to analyze these interactions. This paper builds on these foundations, aiming to rigorously characterize the coupling between SFT and RL, and to derive optimal training strategies.

Practitioners often observe that applying RL after SFT can degrade performance, and vice versa, but lack a theoretical explanation. The core challenge is understanding whether these processes can be decoupled or if their interaction is fundamental. Without this understanding, designing effective training pipelines becomes guesswork, risking capability loss or inefficient resource use. The problem is compounded by the complex, non-convex landscapes of neural networks, where gradient interactions are poorly understood. Addressing this gap is crucial for developing principled methods that balance reward alignment with knowledge retention, especially as models grow larger and more capable.

The paper's key innovations include: 1) Formal proof that SFT and RL are inherently coupled, invalidating the common assumption of decoupling; 2) Derivation of bounds on the degradation of SFT loss and reward gains, showing quadratic scaling with RL duration; 3) Introduction of spectral concentration analysis to quantify gradient orthogonality, explaining why gradients tend to be nearly orthogonal in high dimensions; 4) Explicit formula for the optimal RL stopping time, balancing reward improvement and performance loss. These contributions provide a rigorous theoretical framework that surpasses prior empirical or heuristic approaches.

• �� Model the post-training process as a sequence of SFT and RL steps, analyzing their impact on loss and reward.
• �� Use KL divergence and PL conditions to bound the change in SFT loss after RL, proving quadratic growth.
• �� Apply spectral concentration theory to analyze gradient orthogonality, explaining near-orthogonality in high-dimensional parameter spaces.
• �� Derive the optimal RL duration by balancing the quadratic increase in SFT loss against linear reward gains.
• �� Validate theoretical results with experiments on Qwen3-0.6B, measuring cross-entropy loss and reward under different RL durations.
• �� Use spectral analysis to confirm gradient orthogonality, supporting the theoretical bounds.

Experiments conducted on Qwen3-0.6B model using CoLA dataset, with both SFT-then-RL and RL-then-SFT pipelines. Metrics include cross-entropy loss and reward scores. Different RL durations tested to observe performance degradation and reward decline. Results show RL causes over 5% increase in loss and 8% decrease in reward, matching theoretical bounds. Gradient spectral analysis confirms near-orthogonality, supporting the analysis. Ablation studies vary RL duration and learning rates, demonstrating the quadratic vs linear scaling of degradation and reward, respectively.

Empirical results align with theoretical predictions: RL increases SFT loss quadratically, with a clear optimal stopping point; reward gains are linear and limited; spectral analysis reveals near-orthogonal gradients, explaining the coupling. These findings suggest that naive sequential training is suboptimal, and joint or early-stopping strategies are preferable. The derived bounds and thresholds provide practical guidelines for training large models efficiently while maintaining capabilities.

The insights inform training protocols for large language models, emphasizing joint optimization or early stopping to prevent performance degradation. Applicable to AI alignment, safety, and knowledge retention tasks. Industry practitioners can adopt spectral regularization and adaptive stopping to improve model robustness and alignment, especially in safety-critical applications. Long-term, this framework could guide automated training schedules and multi-task learning strategies, enhancing model capabilities while minimizing risks.

The analysis assumes the PL condition, which may not hold in all neural network landscapes. Experimental validation is limited to a specific model and dataset, requiring broader testing. The framework does not explicitly address multi-stage or multi-task training complexities. Future work should extend the theory to more diverse architectures and real-world scenarios, incorporating dynamic training schedules and multi-objective optimization.