Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
This paper compares Merge, Mix RL, and MOPD in multi-domain RLVR, showing performance gaps up to 8.6 points and providing practical guidance for different scenarios.
Key Findings
Methodology
The study employs shared experts and datasets across model scales (4B, 8B) to systematically compare three fusion paradigms—Merge, Mix RL, and MOPD—on a multi-domain benchmark suite. Using algorithms like Group Relative Policy Optimization (GRPO), Low-Rank Adaptation (LoRA), and KL divergence, it analyzes performance metrics, task vector geometries, and training dynamics. The evaluation covers five domains (Math, Science, Coding, Instruction Following, Agent) with detailed performance and capability retention analysis, providing insights into cross-domain relations and the impact of each fusion method.
Key Results
- All three methods improve single-sample accuracy over the base model, with performance differences within 1.4 points on average, but on a specific benchmark, the gap reaches 8.6 points. Merge compresses expert task vectors into one update, suitable when experts already exist; Mix RL adjusts data proportions during joint training for transfer; MOPD maintains domain-specific gains but cannot surpass teachers, ideal when preserving expertise is prioritized.
- Across five domains, reasoning-intensive areas like math, science, and coding reinforce each other, while instruction following and agent use are nearly orthogonal, as shown by task vector geometry. The methods differ in how they transfer or preserve domain knowledge, impacting their effectiveness in various scenarios.
- Training dynamics reveal that Mix RL depends on data mixture proportions, MOPD is limited by teacher performance, and Merge compresses all updates into a single vector. These differences influence their ability to transfer knowledge and maintain capabilities, guiding practical choices based on cost, existing experts, and domain relations.
Significance
This work provides a comprehensive comparison of three major fusion paradigms in multi-domain RLVR, clarifying their performance, cost, and suitability. It advances understanding of how task relations influence transfer and retention, offering a theoretical and practical foundation for building scalable, multi-task language models. The insights into task vector geometry and training dynamics deepen the comprehension of internal model representations, informing future design of multi-domain AI systems with optimized efficiency and capability preservation.
Technical Contribution
The paper introduces a unified framework for comparing Merge, Mix RL, and MOPD, combining task vector geometry and training dynamics analysis. It proposes a low-cost, effective Merge strategy based on task vector compression, a flexible data pooling approach in Mix RL, and a multi-teacher distillation method in MOPD. The systematic evaluation across model scales and benchmarks demonstrates their distinct strengths and limitations, enriching the theoretical landscape of multi-task learning and model fusion.
Novelty
This is the first systematic comparison of the three fusion paradigms within a unified experimental setup, integrating task vector geometry and training dynamics analysis. The innovative task vector compression method for Merge, along with the comprehensive evaluation across multiple domains and model sizes, provides novel insights into the internal mechanisms of multi-task model fusion. These contributions significantly advance the state-of-the-art in multi-domain model consolidation.
Limitations
- The experiments are limited to specific model sizes (4B, 8B) and five predefined domains; broader generalization to larger models and more diverse tasks remains to be validated.
- MOPD’s reliance on teacher models constrains its ability to surpass teachers, limiting its application in scenarios demanding performance beyond the existing experts.
- Training costs, especially for MOPD, remain high, and the efficiency of fusion methods needs further optimization for deployment in resource-constrained environments.
Future Work
Future research should explore scaling fusion methods to larger models and more domains, develop adaptive task vector adjustment mechanisms, and optimize training efficiency. Combining reinforcement learning with self-supervised learning could enhance transfer and generalization. Investigating theoretical limits of fusion and robustness in real-world scenarios will further improve multi-task AI systems, making them more versatile and accessible.
AI Executive Summary
As large language models evolve, their ability to perform multiple tasks across diverse domains becomes increasingly important. Traditional approaches often involve training separate experts for each domain, which is costly and inefficient. To address this, researchers have developed various fusion strategies—Merge, Mix RL, and MOPD—that aim to consolidate domain-specific knowledge into a single model.
This paper systematically compares these three paradigms using shared experts and datasets across different model scales (4B and 8B) on a comprehensive multi-domain benchmark suite. The results reveal that, on average, performance differences are minimal, within 1.4 points; however, on specific benchmarks, the gap can reach 8.6 points, highlighting the importance of domain relations and fusion strategies.
The analysis combines task vector geometry and training dynamics, showing that reasoning domains like math, science, and coding tend to reinforce each other, while instruction following and agent use are nearly orthogonal. Merge compresses expert task vectors into a single update, suitable when experts already exist. Mix RL pools data for joint training, adjusting domain proportions to facilitate transfer. MOPD employs multi-teacher distillation, preserving domain-specific gains but limited by teacher performance.
All methods improve single-sample accuracy without expanding capabilities or losing existing ones. Practical guidelines emerge: use Merge when experts are available; opt for Mix RL when training a unified model without experts; choose MOPD when maintaining domain-specific gains is critical. This work advances understanding of multi-task model fusion, providing valuable insights for deploying scalable, multi-domain AI systems in industry and research.
Deep Analysis
Background
The development of large language models (LLMs) has led to significant progress in AI capabilities, with reinforcement learning with verifiable rewards (RLVR) emerging as a key technique for enhancing specific skills such as math, coding, and reasoning. Early works like Guo et al. (2025) demonstrated the effectiveness of RLVR in domain-specific tasks, but scaling this to multiple domains introduces challenges such as knowledge interference and resource costs. Existing methods include parameter fusion techniques like LoRA, data pooling strategies, and multi-teacher distillation, each with limitations in transferability, cost, or capability retention. Despite advances, a comprehensive understanding of how different fusion paradigms compare across model sizes and tasks remains lacking, hindering practical deployment in multi-task scenarios.
Core Problem
The core challenge lies in efficiently consolidating multiple domain-specific experts into a single, versatile model without sacrificing performance or incurring prohibitive costs. Existing strategies either require extensive retraining, suffer from limited transferability, or fail to preserve domain-specific gains. Determining the optimal fusion method depends on factors like existing expertise, computational resources, and desired capability retention. The lack of a unified evaluation framework complicates decision-making, especially when models are scaled up or applied across diverse tasks. Addressing these issues is crucial for advancing multi-task AI systems capable of handling real-world complexity.
Innovation
This work introduces a systematic comparison of three fusion paradigms—Merge, Mix RL, and MOPD—within a unified framework. It innovates by analyzing task vector geometry to understand domain relationships and how they influence fusion outcomes. The paper proposes a low-cost Merge method based on task vector compression, enabling rapid integration of experts without retraining. It also develops a data pooling approach in Mix RL, adjusting domain proportions to optimize transfer. MOPD leverages multi-teacher distillation to preserve domain-specific knowledge while maintaining efficiency. These innovations collectively provide a comprehensive understanding of the trade-offs and practical considerations in multi-domain model fusion.
Methodology
- �� Construct shared experts for five domains (math, science, coding, instruction, agent) using RLVR on 4B and 8B models.
- �� Implement Merge: combine expert task vectors via linear combination, updating base weights without retraining.
- �� Implement Mix RL: perform joint RLVR training on a mixed dataset with domain proportions (e.g., 25% math, 22% science), enabling cross-domain transfer.
- �� Implement MOPD: distill five expert models into a single student via on-policy KL divergence minimization, maintaining domain-specific outputs.
- �� Evaluate performance across benchmarks (AIME, GPQA, LiveCode, IFBench, BFCL), analyze task vector geometry, and training dynamics to understand transfer and retention effects.
Experiments
The experiments involve training models with shared experts across five domains, comparing fusion strategies on 4B and 8B models. Each method’s performance is assessed using standardized benchmarks, measuring accuracy@16 and capability retention. The analysis includes task vector geometry to interpret domain relations, training dynamics to observe transfer patterns, and cost evaluation to determine efficiency. Hyperparameters such as task vector combination weights, data proportions, and distillation settings are tuned for optimal performance. Ablation studies examine the impact of different fusion parameters and the influence of domain relatedness, providing a comprehensive comparison of the paradigms.
Results
All fusion methods improve single-sample accuracy over the base models, with average gains of 5.3 to 6.7 points on 4B and 10.8 to 12.6 points on 8B. Despite minimal average differences, on specific benchmarks like AIME, the performance gap reaches 8.6 points, emphasizing the importance of domain relations. Merge effectively consolidates expert knowledge, especially for reasoning domains, while Mix RL enhances transfer via data proportions. MOPD maintains domain-specific gains but does not surpass teacher models, constrained by its training objective. Task vector geometry reveals that reasoning domains reinforce each other, whereas instruction and agent domains are orthogonal, influencing fusion outcomes.
Applications
The insights from this study guide the deployment of multi-task language models in real-world applications such as intelligent tutoring, customer support, and automated coding. By selecting appropriate fusion strategies based on existing expertise, resource constraints, and desired capabilities, practitioners can build scalable, efficient, and capable AI systems. The ability to preserve domain-specific gains while enabling cross-domain transfer accelerates the development of versatile AI assistants, reducing costs and improving performance across diverse tasks. Future integration with adaptive fusion mechanisms could further enhance robustness and scalability.
Limitations & Outlook
The study focuses on specific model sizes and a fixed set of five domains, limiting generalization to larger models or more diverse tasks. MOPD’s dependence on teacher models restricts its ability to surpass them, which may hinder applications requiring performance beyond current experts. The computational costs, especially for MOPD’s distillation process, remain high, posing challenges for large-scale deployment. Future work should explore more scalable fusion algorithms, broader domain coverage, and adaptive methods to address these limitations.
Plain Language Accessible to non-experts
Imagine a big kitchen where different chefs specialize in making different dishes—pasta, sushi, desserts, etc. Each chef has their own secret recipes and techniques. Now, if the kitchen wants to create a new chef who can cook all these dishes well, they can do it in three ways: one, they can combine all the recipes into a master recipe (Merge); two, they can let all chefs teach the new chef by sharing their techniques during cooking (Mix RL); or three, they can have each chef guide the new chef step-by-step, making sure he learns each skill properly (MOPD). Each method has pros and cons: combining recipes is quick but might lose some flavor; teaching during cooking is flexible but takes time; guiding step-by-step keeps the skills intact but is costly. The kitchen chooses based on their needs—speed, quality, or cost—to make the best new chef.
ELI14 Explained like you're 14
Imagine you're trying to learn how to do different sports—soccer, basketball, skateboarding—from your friends. One way is to take all their tips and combine them into one big set of instructions (Merge). Another way is to practice with all your friends at once, mixing their advice as you go (Mix RL). The third way is to have each friend teach you step-by-step, making sure you learn each skill properly (MOPD). Each method works differently: combining tips is quick but might miss some details; practicing with everyone helps you learn faster but can be confusing; step-by-step teaching keeps your skills clear but takes longer and costs more. You pick the method based on what you want—speed, quality, or saving energy—and that helps you become good at all sports!
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.