Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
M-GRPO extends Group Relative Policy Optimization for hierarchical multi-agent systems, aligning heterogeneous trajectories to improve reasoning performance.
Key Findings
Methodology
This paper introduces the Multi-Agent GRPO (M-GRPO), an extension of the Group Relative Policy Optimization framework tailored for vertical multi-agent systems. It employs hierarchical credit assignment by computing group-relative advantages for both main and sub-agents, addressing challenges of asynchronous, heterogeneous trajectories and distributed deployment. The approach incorporates a trajectory-alignment scheme that duplicates or drops sub-trajectory samples to maintain fixed batch sizes, facilitating efficient batched policy updates. The training pipeline is distributed: main and sub-agents operate on separate servers, exchanging minimal statistics via shared storage, thus avoiding end-to-end backpropagation issues. The core algorithm involves advantage estimation, clipping, and policy updates, combined with specific reward structures for both main and sub-agents, optimized to enhance multi-turn reasoning and tool use tasks.
Key Results
- On real-world benchmarks GAIA, XBench-DeepSearch, and WebWalkerQA, M-GRPO consistently outperforms single-agent GRPO and frozen sub-agent multi-agent baselines, with performance gains exceeding 10%. The method's trajectory alignment significantly stabilizes training, enabling scalable distributed optimization. Experimental data show improved task success rates, stability, and sample efficiency, validating the approach’s effectiveness in complex reasoning scenarios.
- Ablation studies reveal joint training of main and sub-agents yields approximately 8% higher success metrics than training only the main agent. The trajectory alignment mechanism reduces training variance and enhances generalization, especially in multi-step reasoning and tool invocation tasks.
- Across diverse tasks and data distributions, M-GRPO demonstrates robustness and adaptability, outperforming traditional methods in multi-turn reasoning, tool integration, and complex problem-solving.
Significance
This work addresses critical challenges in training heterogeneous multi-agent systems, notably trajectory heterogeneity and distributed deployment. By enabling scalable, stable, and efficient training, it advances the development of autonomous, collaborative AI agents capable of complex reasoning. The hierarchical framework and trajectory alignment provide a new paradigm that bridges the gap between theoretical reinforcement learning and practical multi-agent deployment, with broad implications for AI research and industry applications such as automated reasoning, knowledge retrieval, and multi-modal interaction.
Technical Contribution
Technically, the paper extends GRPO to a hierarchical multi-agent setting, introduces a trajectory-alignment scheme for variable sub-agent invocations, and implements a distributed training pipeline that decouples agent updates across servers. The advantage estimation and policy clipping mechanisms ensure stable optimization despite asynchronous, heterogeneous trajectories. This approach enables role-specific LLM training, improves scalability, and enhances the stability and performance of multi-agent reinforcement learning systems.
Novelty
This is the first work to integrate hierarchical multi-agent architectures with group-relative advantage estimation, specifically addressing asynchronous, heterogeneous trajectory alignment in distributed training. Unlike prior methods that share parameters across agents, this approach allows role-specific LLMs and decoupled optimization, significantly improving training stability and scalability in complex reasoning tasks.
Limitations
- The fixed trajectory alignment target d may limit flexibility in highly heterogeneous or dynamic environments, requiring adaptive mechanisms.
- Shared storage reliance introduces potential bottlenecks and synchronization overhead, especially at larger scales.
- Computational costs remain high due to multiple rollouts and distributed training, posing challenges for real-time deployment.
Future Work
Future directions include developing adaptive trajectory alignment strategies, integrating meta-learning for better parameter initialization, and extending the framework to more complex, multi-layered multi-agent systems. Exploring self-supervised signals and reducing computational overhead are also promising avenues to enhance scalability and robustness.
AI Executive Summary
Multi-agent systems have long been recognized for their potential to tackle complex reasoning tasks through collaboration. However, training such systems remains challenging, especially when agents are heterogeneous, operate asynchronously, and are deployed across distributed servers. Existing approaches often rely on shared models or synchronous training, which limit scalability and stability.
This paper introduces M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed specifically for vertical multi-agent architectures. The core innovation lies in the trajectory-alignment scheme, which ensures fixed batch sizes despite variable sub-agent invocations. By duplicating or dropping sub-trajectory samples, the method maintains consistent tensor shapes, enabling efficient batched policy updates. The distributed training pipeline allows main and sub-agents to operate on separate servers, exchanging minimal statistics via shared storage, thus avoiding the complexities of end-to-end backpropagation.
Experimental results on benchmarks such as GAIA, XBench-DeepSearch, and WebWalkerQA demonstrate that M-GRPO consistently outperforms traditional single-agent and frozen sub-agent baselines. The improvements include over 10% in task success rates, enhanced training stability, and better sample efficiency. Ablation studies confirm that joint training of main and sub-agents yields significant performance gains, validating the importance of the proposed trajectory alignment and distributed optimization.
Overall, this work advances the state-of-the-art in multi-agent reinforcement learning by providing a scalable, stable, and effective training framework. It opens new avenues for deploying autonomous, tool-augmented reasoning agents in real-world applications, from complex knowledge retrieval to multi-modal decision-making. Future research will focus on adaptive alignment strategies, reducing computational costs, and extending the framework to more complex multi-layered systems, promising a significant leap forward in AI collaboration capabilities.
Deep Dive
Abstract
Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks.