One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
This study reveals simulator collapse in multi-agent RL, proposing Verbalized Sampling and Co-Training to enhance generalization, improving success rates by up to 14%.
Key Findings
Methodology
This paper formalizes the phenomenon of simulator collapse from a theoretical perspective, defining mode collapse as the tendency of language models (LLMs) to respond predominantly in a narrow set of modes. It analyzes how training against a fixed, mode-collapsed simulator biases the policy gradient toward exploiting this limited response pattern, leading to overfitting and poor transferability. To address this, two complementary methods are proposed: Verbalized Sampling, which during inference samples responses from a verbalized response distribution to increase diversity; and Co-Training, which jointly trains the policy and a trainable simulator, allowing the simulator to evolve dynamically during training. Theoretical bounds are derived to quantify the bias introduced by mode collapse, and extensive experiments on three benchmarks—Persuasion for Good, τ²-bench, and CooperBench—demonstrate the effectiveness of these methods. Results show success rate improvements of up to 9% with Verbalized Sampling and 14% with Co-Training, with additional validation from human studies confirming enhanced real-world robustness.
Key Results
- In Persuasion for Good, single-simulator RL peaks early with success rates around 65-70%, then collapses, whereas Verbalized Sampling boosts success to 74%, and Co-Training further elevates it to 82%.
- On τ²-bench, success rates for single-simulator RL decline from 55% to 45%, while Verbalized Sampling achieves 54%, and Co-Training reaches 58%, indicating significant generalization gains.
- Human user evaluations reveal that models trained with Co-Training produce more natural and diverse responses, increasing success rates by approximately 12% over traditional RL, confirming practical benefits.
Significance
This work addresses a fundamental challenge in multi-agent RL: the tendency of models to overfit narrow response modes when trained against fixed simulators. By formalizing the problem and proposing solutions that promote environment diversity, the study advances the development of more robust, generalizable dialogue systems. The findings have broad implications for deploying AI in real-world human-AI interaction scenarios, such as customer service, collaborative coding, and tutoring, where response variability and adaptability are critical. The theoretical insights and practical algorithms pave the way for future research on environment-aware training paradigms, ultimately contributing to AI systems capable of handling the complexity and unpredictability of real-world interactions.
Technical Contribution
The paper's core technical contribution lies in the formal analysis of simulator collapse, deriving bounds on the bias introduced into policy gradients by mode-concentrated response distributions. It introduces Verbalized Sampling as an inference-time method to approximate the true response distribution, thereby restoring diversity without retraining. Additionally, the Co-Training framework jointly optimizes the policy and a trainable simulator, enabling the environment to adapt dynamically during training, which prevents the policy from overfitting to a static mode. The authors also release SCOPE, an open-source toolkit that facilitates implementation of population co-training, self-play, and multi-model rotation, fostering further research and application development. The combination of theoretical guarantees and empirical validation distinguishes this work as a significant step forward in addressing environment-induced overfitting in multi-agent RL.
Novelty
This research is pioneering in its systematic formalization of simulator collapse as a structural failure mode in multi-agent RL. The introduction of Verbalized Sampling and Co-Training as solutions—particularly the latter's dynamic environment adaptation—represents a novel approach that departs from traditional static environment training. Unlike prior work that focuses solely on policy architectures or reward shaping, this study emphasizes environment diversity as a critical factor for generalization, supported by rigorous theoretical bounds and extensive experiments. The open-source SCOPE framework further enhances its novelty by providing a practical platform for community adoption and extension.
Limitations
- Despite improvements, the methods may still struggle in scenarios where the simulator's response distribution is extremely narrow or heavily biased, limiting diversity gains.
- Co-Training requires additional computational resources and careful reward design to ensure the simulator's responses evolve meaningfully during training, which may pose challenges in large-scale applications.
- The approach's effectiveness in high-dimensional, real-world tasks with complex behaviors remains to be fully validated, necessitating further research to scale these techniques.
Future Work
Future research should explore integrating multi-modal data and meta-learning techniques to further enhance environment diversity. Developing adaptive reward schemes that dynamically balance response variability and realism could improve training stability. Extending these methods to large-scale, real-world human-AI interaction systems, such as customer service bots or educational tutors, will be crucial. Additionally, investigating how human feedback can be incorporated into the environment evolution process promises to make AI systems more aligned with genuine user behaviors, ultimately pushing towards truly autonomous and robust multi-agent dialogue systems.
AI Executive Summary
Multi-agent reinforcement learning (MARL) has become a cornerstone in advancing human-AI interaction, particularly in dialogue systems, collaborative coding, and tutoring. However, a persistent challenge has hindered progress: the tendency of models to overfit to narrow, mode-collapsed simulators during training. This phenomenon, termed simulator collapse, occurs when large language models (LLMs) respond predominantly in a limited set of behaviors, which biases the policy gradient toward exploiting these narrow responses. As a result, policies trained against such simulators perform well in in-distribution settings but fail to generalize to unseen environments or real users, severely limiting practical deployment.
Recognizing this problem, the authors formalize the concept of simulator collapse through a rigorous theoretical framework. They demonstrate that mode collapse biases the gradient toward a deterministic mode response, reducing policy entropy and leading to overfitting. To counteract this, they propose two complementary solutions: Verbalized Sampling, which during inference samples responses from a verbalized distribution to increase diversity, and Co-Training, which during training jointly optimizes the policy and a trainable simulator, allowing the environment to evolve dynamically. These methods aim to prevent the policy from overfitting to a narrow response mode, thereby enhancing its robustness and generalization.
Extensive experiments across three benchmark tasks—Persuasion for Good, τ²-bench, and CooperBench—validate the effectiveness of these approaches. Verbalized Sampling improves success rates by up to 9%, while Co-Training achieves improvements up to 14%. Human studies further confirm that policies trained with these methods perform better in real-world interactions, exhibiting higher naturalness and robustness. The results underscore the importance of environment diversity, not just policy design, for achieving reliable multi-turn RL in practical applications.
The study's theoretical insights and practical algorithms mark a significant advance in addressing environment-induced overfitting. By releasing the open-source SCOPE framework, the authors facilitate community adoption and further research. Despite these advances, challenges remain in scaling these techniques to highly complex, high-dimensional tasks and ensuring the simulated responses accurately reflect real user behaviors. Future work will likely explore integrating multi-modal data, adaptive reward schemes, and human-in-the-loop feedback to further improve environment diversity and policy robustness. Overall, this research charts a promising path toward more generalizable, resilient AI systems capable of engaging in natural, multi-turn human interactions across diverse scenarios.
Deep Analysis
Background
多智能体强化学习（MARL）在自然语言处理、机器人协作和自动客服等领域取得了显著进展。早期研究多集中于单轮任务，结合深度强化学习（Deep RL）和大语言模型（如GPT-3、Qwen系列）实现策略优化。代表性工作包括RLHF（Reinforcement Learning from Human Feedback）和模仿学习方法，极大提升了模型的生成质量和交互能力。随着任务复杂度提升，多轮对话成为研究焦点，诸如Persuasion for Good、τ²-bench和CooperBench等数据集推动了多轮对话系统的发展。这些方法在训练效率和性能指标上取得了突破，但在泛化能力方面仍存在瓶颈。传统训练依赖固定模拟器，忽视了真实用户行为的多样性，导致策略在实际环境中表现不佳。近年来，模拟器崩溃（mode collapse）问题逐渐引起关注，学界开始探索多样性增强、联合训练等新技术，试图突破这一限制，为实现更自然、更鲁棒的对话系统提供理论基础。
Core Problem
当前多轮对话强化学习普遍依赖单一、固定的模拟器，导致训练出的策略过度拟合模拟器的狭窄响应模式。这种模式崩溃现象使得策略在面对未见环境和真实用户时表现不佳，严重制约了模型的泛化能力。具体而言，模拟器响应趋向于高概率的模式，忽略了用户行为的多样性，训练过程中策略逐渐偏向狭窄的应答策略，最终在实际应用中崩溃。解决这一问题的难点在于如何在保证训练效率的同时，增加模拟环境的多样性，避免策略陷入局限。
Innovation
本研究的核心创新在于：1）系统定义了模拟器崩溃的机制，揭示了模式崩溃对梯度偏置的影响，为理论分析提供基础；2）提出Verbalized Sampling，通过采样多样化响应，缓解模拟器响应的模式崩溃问题，增强环境多样性；3）引入Co-Training，联合优化策略和模拟器，使模拟器响应在训练中不断演变，避免固定模式的陷阱。这些创新突破了传统单模态模拟器的局限，为多轮对话RL的泛化提供了新思路。
Methodology
- �� 模型框架：采用Partially Observable Markov Decision Process（POMDP）模型，定义状态、动作、奖励，结合深度RL优化策略。• 模拟器定义：利用大语言模型（如GPT-3.5、Qwen3-4B）作为模拟用户，响应分布ϕψ(·|st, aπt)，在训练中保持固定。• 模式崩溃分析：通过理论推导，定义响应的最可能行为（mode）a⋆ϕ，分析响应偏向性ϵϕ，揭示其对梯度偏置的影响。• 解决方案一（推理阶段）：Verbalized Sampling，利用响应分布采样多个响应，丰富模拟环境，减少模式崩溃。• 解决方案二（训练阶段）：Co-Training，联合优化策略和模拟器，使模拟器响应随训练动态演变，避免固定响应模式。• 理论保证：推导梯度偏差界限，说明模拟器崩溃导致的偏向性。• 实验设计：在三个任务中比较单模拟器RL、Verbalized Sampling和Co-Training的性能，验证理论预测。
Experiments
- �� 数据集：Persuasion for Good、τ²-bench和CooperBench，涵盖劝说、协作编码等多轮对话任务。• 实验流程：在每个任务中，训练三种模型（单模拟器RL、Verbalized Sampling、Co-Training），在训练集和未见模拟器的测试集上评估成功率、策略熵和人类评价指标。• 超参数：采用Qwen3-4B和Qwen3.5-9B模型，训练步数达25000步，调节采样温度以控制响应多样性。• 评估指标：成功率、策略熵、泛化能力（在未见模拟器上的表现）以及人类交互中的自然度。• 附加分析：通过消融实验验证Verbalized Sampling和Co-Training的贡献，分析模拟器响应多样性对策略性能的影响。
Results
- �� 单模拟器RL在训练后期表现出明显的崩溃趋势，成功率在Persuasion for Good中从最高的85%下降到65%，在未见模拟器上表现更差。• 引入Verbalized Sampling后，成功率提升至74%，比传统RL高出9%；Co-Training进一步提升至82%，表现出更强的泛化能力。• 策略熵在单模拟器RL中迅速下降至接近零，反映出策略过度拟合狭窄响应；而两种新方法保持较高的策略多样性，增强了模型的鲁棒性。• 人类用户交互测试中，采用Co-Training的模型在自然度和应答多样性方面优于传统模型，成功率提升约12%。
Applications
- �� 立即应用：在客服机器人、智能助理等场景中，利用多样化模拟环境训练更具泛化能力的对话策略，提升用户体验和系统鲁棒性。• 长远目标：构建具有自主学习和适应能力的多智能体系统，能在复杂、多变的真实环境中持续优化，减少对人工调节的依赖，实现真正的自主交互系统。
Plain Language Accessible to non-experts
想象你在一家厨房里做饭，厨房里有一个厨师（模拟器）负责给你提供食材和建议。以前，这个厨师总是用一种固定的方式推荐菜谱，比如只推荐意大利面。你学会了只用意大利面做饭，但当你换到别的厨房或面对不同的客人时，就会发现这些菜谱不适用。这就像模拟器崩溃——厨师只会一种回答，导致你做的菜不够多样，也不能应对不同的情况。为了改善这个问题，你可以让厨师在推荐菜谱时考虑更多的可能性，比如问“你还会推荐什么？”这样，他就会给出更多不同的建议。或者，你和厨师一起学习，厨师随着你的练习不断改变他的推荐方式。这样，无论面对什么样的客人，你都能做出合适的菜肴。这就像Verbalized Sampling和Co-Training，让厨房变得更灵活多样，能应对各种不同的需求。
ELI14 Explained like you're 14
想象你在玩一个游戏，你的角色需要和很多不同的NPC（非玩家角色）对话。以前，你用一种固定的对话套路，比如只说“你好”，然后他们就会一直回答一样的话。这就像模拟器只会一种回应，导致你学会了一套死板的对话技巧。当你遇到真正的玩家或不同的NPC时，这些套路就不管用了。为了让你的对话更自然，你可以让NPC在每次回答时考虑多种可能，比如问“你还想聊点别的吗？”这样他们就会给出不同的回答。或者，你和NPC一起学习，让他们随着你的对话不断变化，变得更聪明、更有趣。这样，无论面对谁，你都能有流畅自然的对话。这就像Verbalized Sampling和Co-Training，让AI变得更灵活，能应对各种不同的场景，不再死板。
Glossary
Mode Collapse (模式崩溃)
指大语言模型在模拟用户行为时响应集中在少数几种模式，缺乏多样性，导致训练策略过度拟合狭窄响应。
论文中分析模拟器响应的模式集中现象，影响策略的泛化能力。
Verbalized Sampling (语言化采样)
在推理阶段，通过采样模拟器响应的多样化分布，增加环境多样性，缓解模式崩溃问题。
作为解决模拟器崩溃的方案之一，用于丰富模拟环境。
Co-Training (联合训练)
在训练阶段同时优化策略和模拟器，使模拟器响应随训练演变，避免固定模式。
通过联合优化，提升策略在未见环境中的泛化能力。
POMDP (部分可观测马尔可夫决策过程)
一种模型，用于描述多轮对话中，状态部分可观测，决策依赖历史信息。
论文中用以建模多轮对话的环境。
Mode (模式)
模拟器响应中概率最高的行为类型，是模型偏向的主要响应。
分析模拟器响应偏向的核心概念。
Policy Entropy (策略熵)
衡量策略输出多样性的指标，熵越低表示策略越单一。
训练中观察到的策略崩溃表现之一。
Gradient Bias (梯度偏差)
由模拟器模式崩溃引起的，策略梯度偏向狭窄响应的偏差。
理论分析模拟器崩溃对训练的影响。
Response Distribution (响应分布)
模拟器在给定状态下可能响应的概率分布。
Verbalized Sampling采样的基础。
Policy Diversity (策略多样性)
策略输出的多样性，反映模型应对不同场景的能力。
避免模式崩溃的关键指标。
Training Environment Diversity (训练环境多样性)
训练中模拟环境的多样性，确保策略的泛化能力。
论文强调的核心观点。
Open Questions Unanswered questions from this research
- 1 尽管提出了Verbalized Sampling和Co-Training，但在极端偏狭或偏离真实用户行为的模拟器上效果仍有限，未来需探索更强的多样性增强机制。
- 2 如何在大规模真实用户交互中高效训练多智能体系统，减少训练成本，提升系统鲁棒性，仍是未解决的难题。
- 3 模拟器响应的多样性与真实性之间的平衡，以及如何在保证多样性的同时保持响应的真实性，是未来研究的重要方向。
- 4 现有方法主要在有限任务和数据集上验证，推广到更复杂、更高维的实际场景仍需大量工作。
- 5 如何结合人类反馈机制，动态调整模拟器响应策略，以更贴近真实用户行为，也是未来的研究重点。
Abstract
Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $τ^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.
References (20)
Natural Emergent Misalignment from Reward Hacking in Production RL
M. MacDiarmid, Benjamin Wright, Jonathan Uesato et al.
Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Jiayi Zhang, Simon Yu, Derek Chong et al.
τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Victor Barres, Honghua Dong, Soham Ray et al.
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Xuhui Zhou, Weiwei Sun, Qianou Ma et al.
KL-Regularized Reinforcement Learning is Designed to Mode Collapse
Anthony GX-Chen, Jatin Prakash, Jeff Guo et al.
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Liwei Jiang, Yuanjun Chai, Margaret Li et al.
TOM-SWE: User Mental Modeling For Software Engineering Agents
Xuhui Zhou, Valerie Chen, Z. Wang et al.
UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
Cheng Qian, Zuxin Liu, Akshara Prabhakar et al.
CooperBench: Why Coding Agents Cannot be Your Teammates Yet
A. Khatua, Hao Zhu, Peter Tran et al.
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Xuewei Wang, Weiyan Shi, Richard Kim et al.
SGLang: Efficient Execution of Structured Language Model Programs
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie et al.
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison et al.
LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
J. Park, Carolyn Q. Zou, Jonne Kamphorst et al.
NoveltyBench: Evaluating Language Models for Humanlike Diversity
Yiming Zhang, Harshita Diddee, Susan Holm et al.
Dota 2 with Large Scale Deep Reinforcement Learning
Christopher Berner, Greg Brockman, Brooke Chan et al.
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
Joseph Suh, Ayush Raj, Minwoo Kang et al.
Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
Marwa Abdulhai, Ryan Yi Cheng, D. Clay et al.
Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
Yanming Wan, Jiaxing Wu, Marwa Abdulhai et al.
The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models
Yunze Xiao, Vivian Zhang, Chenghao Yang et al.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
M. Shoeybi, M. Patwary, Raul Puri et al.