SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
This study analyzes SFT and RL in multi-task learning, revealing RL induces sparse, orthogonal parameter updates, reducing task interference.
Key Findings
Methodology
Using combined empirical and theoretical analysis, the study demonstrates RL causes sparse, nearly orthogonal parameter updates across tasks, based on gradient interference theory. It compares parameter changes, gradient inner products, and interference bounds between SFT and RL, revealing their mechanistic differences. The analysis involves advantage functions, policy sources, and high-dimensional probability principles to explain the observed phenomena.
Key Results
- Experiments show RL induces parameter updates with average magnitude 0.02, over 100 times smaller than SFT's 0.74. Gradient cosine similarity between tasks in RL is near zero (~10^-5), while SFT's is ~10^-1, causing interference.
- In multi-stage training, SFT performance drops by 23.1%, whereas RL achieves a 24.9% improvement. Single-task training results indicate SFT improves target tasks by 4% but drops 5.1% on others; RL improves target by 6.8% and boosts untrained tasks by 2.3%.
- Theoretically, SFT interference is norm-limited, driven by absolute gradient magnitude, while RL interference is variance-limited, bounded by intra-group gradient variance due to advantage normalization and on-policy sampling.
- Proposes Parallel-RL, training tasks independently then merging updates, which maintains performance and improves efficiency, validated through extensive experiments.
Significance
This work advances understanding of multi-task training by revealing the fundamental mechanisms enabling task coexistence in RL. It addresses the longstanding challenge of gradient interference in large models, offering a theoretical basis and practical framework (Parallel-RL) for stable, efficient multi-task learning. The insights could transform how models are trained for diverse real-world applications, balancing performance and computational cost, and paving the way toward more robust artificial general intelligence.
Technical Contribution
The paper introduces a gradient interference framework distinguishing norm-limited SFT from variance-limited RL. It provides theoretical bounds based on advantage functions and policy sources, demonstrating how RL's on-policy nature and advantage normalization lead to near-orthogonal, sparse updates. The Parallel-RL paradigm operationalizes these insights, enabling task decoupling and efficient multi-task training, a significant step beyond existing methods.
Novelty
This is the first comprehensive analysis linking gradient interference mechanisms to the empirical stability of RL in multi-task learning. It uncovers the core reason behind RL’s task coexistence—its sparse, orthogonal parameter updates—supported by rigorous theoretical bounds. The Parallel-RL framework innovatively leverages these properties, offering a practical solution for scalable multi-task training in large models.
Limitations
- The analysis primarily focuses on gradient norms and variance bounds, not fully accounting for large-scale model training costs or real-world data complexities.
- While promising, the robustness of Parallel-RL in highly heterogeneous or adversarial task environments remains to be validated.
- The approach assumes certain properties of advantage functions and policy sampling, which may not hold universally across all RL algorithms or model architectures.
Future Work
Future research will explore adaptive task selection, finer-grained decoupling strategies, and integration with meta-learning to further improve multi-task robustness. Extending theoretical bounds to more complex RL algorithms and larger models, as well as testing in industrial scenarios, will be key steps toward practical deployment.
AI Executive Summary
Multi-task learning in large language models faces a fundamental challenge: task interference that hampers performance and stability. Traditional supervised fine-tuning (SFT) often suffers from severe conflicts when trained across multiple tasks sequentially, leading to catastrophic performance drops. In contrast, reinforcement learning (RL) demonstrates a remarkable ability to enable models to learn multiple tasks simultaneously without interference, maintaining stable performance across stages.
This study combines empirical observations with rigorous theoretical analysis to uncover the underlying mechanisms. It reveals that RL induces sparse, nearly orthogonal parameter updates, significantly reducing gradient interference. The core reason lies in the advantage function and on-policy sampling, which constrain the gradient variance and promote task-specific, low-overlap updates. The authors derive bounds showing SFT interference scales with gradient norm, while RL interference is limited by gradient variance, explaining RL’s robustness.
Building on these insights, the paper introduces Parallel-RL, a paradigm that trains tasks independently and merges updates, effectively decoupling task-specific parameters. Experimental results demonstrate that Parallel-RL achieves performance comparable to single-task training while greatly improving training efficiency and stability. This approach opens new avenues for scalable, multi-task large model training, addressing a key bottleneck in AI development.
However, challenges remain, including the computational costs of on-policy RL and the need to validate the approach in diverse, real-world scenarios. Future work will focus on refining task decoupling techniques, extending theoretical guarantees, and applying these methods to industrial-scale models. Overall, this research provides a profound understanding of gradient interference mechanisms, offering a practical framework to realize robust, efficient multi-task learning in large language models, pushing the frontier toward artificial general intelligence.
Deep Dive
Abstract
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.