MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games

Recent advances in large language models (LLMs) have achieved remarkable performance on static benchmarks, but their application in multi-turn, multi-agent environments reveals significant instability. Early decision errors tend to compound over multiple rounds, especially in complex strategic interactions like negotiation or imperfect information games. Traditional prompt engineering techniques, such as chain-of-thought prompting, are static and do not adapt to emergent strategies. Automatic prompt tuning methods, though dynamic, lack persistent memory, leading to high variance and poor reproducibility. Reinforcement learning approaches can optimize strategies but require extensive samples and often remain unstable in multi-agent settings. This context underscores the need for methods that can dynamically adapt and retain knowledge across interactions, improving both robustness and efficiency.

The core challenge is to stabilize multi-turn multi-agent interactions where small early deviations lead to divergent trajectories, biasing win-rate estimates and impairing model rankings. Existing methods either lack memory mechanisms or are computationally expensive. The problem is compounded by the path-dependent nature of these interactions, where rare states and strategic shifts are critical but often underexplored. Achieving reliable, sample-efficient optimization of inference context without model weight updates remains an open challenge, especially for complex, uncertain environments. Addressing this gap is vital for deploying dependable AI agents in real-world scenarios requiring long-term strategic reasoning.

MEMO introduces several key innovations: 1) a persistent memory bank that stores structured insights from self-play trajectories, enabling cross-episode knowledge reuse; 2) a reflection mechanism that distills trajectories into actionable insights, improving context quality; 3) tournament-style context evolution guided by TrueSkill, which models uncertainty and selects stable, high-performing prompts; 4) prioritized replay that revisits rare, decisive states to enhance exploration. Unlike static prompt tuning or RL, MEMO combines memory, reflection, and exploration in a unified framework, allowing continuous, data-efficient optimization without weight updates. This approach significantly reduces variance and improves robustness in multi-agent interactions.

• �� Reflection and memory: After each self-play, the model analyzes trajectories to extract structured insights, stored in a persistent memory bank. • Context evaluation: A candidate pool of prompts is evaluated via self-play, with TrueSkill modeling skill and uncertainty, selecting prompts with high lower-confidence bounds. • Context evolution: New candidates are generated through random proposals and memory-augmented updates, balancing exploration and exploitation. • Trajectory replay: Key states are stored in a buffer, prioritized by rarity, and replayed to reinforce learning. • Iterative optimization: Multiple generations refine prompts, with the best context selected at the end. • Metrics: Win rates, variance, and stability are measured across 50 games per task, with hyperparameters tuned for efficiency.

Experiments involve five text-based games, including negotiation, Kuhn poker, and imperfect/perfect information games. Models evaluated include GPT-4o-mini and Qwen-2.5-7B-Instruct. Each method undergoes multiple runs with fixed self-play budgets, measuring average win rate and variance. Baselines include static prompts, prompt optimization methods (TextGrad, MIPRO, GEPA), and RL approaches (UnstableBaselines, SPIRAL). Key hyperparameters include candidate pool size (8), generations (5), and replay buffer capacity (100,000). Ablation studies isolate the effects of memory and exploration modules. Results show MEMO consistently outperforms baselines, with significant improvements in stability and sample efficiency.

MEMO achieves a mean win rate of 49.5% on GPT-4o-mini, nearly doubling the baseline, with a relative standard error of 6.4%, far lower than other methods. On Qwen, win rate reaches 44.3%, with variance reduced sevenfold. The approach requires only 2000 self-play games per task, compared to RL methods needing 38,000. Ablation studies confirm that both persistent memory and exploration are essential; removing either reduces performance. The structured insights stored in memory enable transferability across tasks, demonstrating strong generalization. Token efficiency is also improved, with MEMO using about a quarter of the tokens of other methods while maintaining or surpassing their performance.

This framework can be applied to dialogue systems, negotiation agents, and strategic decision-making tools, especially where stability and sample efficiency are critical. It supports deploying autonomous agents capable of long-term reasoning, adaptation, and robustness in uncertain environments. The approach reduces reliance on extensive retraining, enabling rapid adaptation to new scenarios. In industry, MEMO can enhance virtual assistants, automated customer service, and multi-agent coordination in complex systems, providing more reliable and interpretable AI behaviors.

The effectiveness depends on the quality of reflection prompts; biased or inaccurate insights can impair learning. Memory growth may become inefficient in very long or complex tasks, requiring better compression. The method's reliance on structured insights limits its applicability in highly dynamic environments with rapid strategy shifts. Additionally, current experiments focus on text-based games; extending to multimodal or real-world scenarios poses challenges. Future work should address these limitations by improving memory management, reflection quality, and scalability.