Constitutional Arms Races in the Public Goods Game: Co-Evolving LLM Constitutions Under Cooperation-Defection Pressure

Large language models (LLMs) have revolutionized natural language processing, enabling increasingly autonomous intelligent agents. Constitutional AI (CAI) leverages human-written principles to align LLM behavior, improving safety and helpfulness in single-agent contexts. However, real-world multi-agent systems involve complex interactions including negotiation, competition, and information sharing, where cooperative assumptions break down. Prior work by Kumar et al. demonstrated that LLM-guided evolutionary search can discover effective cooperative constitutions in multi-agent grid-worlds, outperforming hand-crafted baselines. Yet, these studies focused on static cooperative environments, leaving open questions about constitutional evolution under adversarial pressure, the reliability of LLM mutation operators in such contexts, and the influence of different social dilemma structures. Addressing these gaps is critical as multi-agent AI systems are increasingly deployed in settings with conflicting goals and strategic incentives.

The core problem is designing and evolving natural-language constitutions that govern multi-agent behavior under adversarial pressure, enabling stable and interpretable governance in environments with cooperation-defection tensions. Key challenges include: 1) ensuring the fitness function induces genuine adversarial selection pressure rather than independent faction optimization; 2) verifying the reliability and stability of LLM mutation operators when optimizing adversarial-specialist objectives; 3) understanding how different social dilemma structures (e.g., Public Goods Game vs. spatial grid-world) affect constitutional evolution; and 4) controlling evaluation budget and fitness estimation noise to prevent mode regression and maintain robust search dynamics. Solving these challenges is essential for advancing multi-agent alignment beyond cooperative settings.

This work introduces several core innovations:

1) An LLM-guided evolutionary search framework that co-evolves priority-ordered natural-language constitutions for two adversarial factions, simulating a constitutional arms race, contrasting prior cooperative-only approaches.

2) The formulation and empirical validation of score-advantage fitness (S_own - S_opp) as a necessary mechanism to induce fitness coupling and genuine adversarial dynamics, overcoming the failure of independent scoring.

3) Identification of evaluation seed count K as a critical hyperparameter controlling mode regression in pure adversarial search, highlighting evaluation budget as a key lever alongside mutation operator design.

4) Comprehensive experimentation across structurally distinct environments (PGG and grid-world), demonstrating the generality of the approach and the influence of environmental payoff structure on evolutionary dynamics.

5) Generation of interpretable, transferable adversarial constitutions that serve as red-team artifacts, enabling rigorous robustness testing of cooperative governance mechanisms.

• �� Environment Setup: Two environments are used—Public Goods Game (PGG) with 6 agents (3 Blue cooperators, 3 Red free-riders), 20 rounds, multiplier m in {1.2,1.5,2.0,3.0}; and an 8×8 spatial grid-world with two hidden factions each controlling 3 agents.

• �� Constitutional Representation: Each faction’s constitution is a priority-ordered set of natural-language rules that agents follow verbatim during decision-making.

• �� Evolutionary Framework: OpenEvolve combined with MAP-Elites is employed to perform LLM-guided mutation and selection, alternating updates between Blue and Red factions to maximize fitness.

• �� Fitness Functions: Three modes are explored—faction-specific score Sfaction, score-advantage Sfaction - Sopp, and pure adversarial 1 - Sopp. PGG’s shared pool naturally couples payoffs; grid-world experiments require score-advantage fitness to induce adversarial pressure.

• �� Evaluation Strategy: Multiple random seeds (K=2 or 5) are used to estimate fitness, mitigating noise and preventing mode regression.

• �� Experimental Protocol: Runs span 30 generations, tracking score trajectories, equilibrium convergence, information asymmetry effects, and defensive mechanism impacts.

Experiments are organized into PGG and grid-world categories. In PGG, initial Blue score is 0.370 and Red 0.177; after 30 generations, both converge near 0.78, robust across multipliers m. Pure adversarial fitness experiments show Red can suppress Blue’s score effectively. Grid-world experiments reveal independent scoring fails to induce adversarial dynamics; adopting score-advantage fitness restores competition. Fixing Blue’s cooperative constitution C* and evolving Red from a zero-sum seed yields Red advantage averaging -0.27, indicating structural resilience. Introducing coordinated attack requirements reduces Red’s advantage to -0.66. Information asymmetry experiments show Red’s advantage increases to +0.415 when observing Blue’s actions. Pure adversarial search with K=2 suffers mode regression; increasing to K=5 stabilizes specialization, underscoring evaluation budget’s importance.

In PGG, Blue’s score rose from 0.370 to 0.777 and Red’s from 0.177 to 0.782 over 30 generations, converging to a stable near-parity equilibrium (~0.78) robust across multipliers m. Independently scored environments showed negligible correlation between faction scores (corr=+0.088), failing to produce adversarial pressure; score-advantage fitness restored competitive dynamics. Pure adversarial fitness experiments revealed evaluation seed count K as a pivotal factor: K=2 led to mode regression, whereas K=5 maintained stable adversarial specialization throughout 30 generations. Information asymmetry experiments demonstrated Red’s significant advantage (+0.415) when observing Blue’s actions. Defensive mechanisms requiring coordinated attacks effectively reduced Red’s mean advantage from -0.27 to -0.66, illustrating practical mechanism design levers.

The findings apply to governance rule design in multi-agent systems involving cooperation and competition, such as automated economic markets, distributed resource management, and security red teaming. The evolved natural-language constitutions are interpretable, facilitating human expert review and adjustment to enhance transparency and safety. Red constitutions serve as red-team artifacts for robustness testing of cooperative mechanisms, aiding secure deployment and regulation of multi-agent AI systems.

Fitness estimation noise is heavily dependent on evaluation seed count; low budgets cause mode regression, limiting search efficiency and stability. Experiments are limited to PGG and a specific grid-world, lacking validation in more complex or real-world multi-agent adversarial scenarios, thus limiting generalizability. No transfer experiments comparing adversarially-evolved constitutions against novel adversaries were conducted, leaving open questions about practical robustness superiority.