SPADE: Self-Play in Adaptive Synthetic Executable Environments
SPADE employs self-play with a single LLM generating and optimizing executable environments, achieving an average 5.3% improvement across benchmarks.
Key Findings
Methodology
The SPADE framework leverages a large language model (LLM) to perform dual roles: as an environment designer and as a reasoning agent. The environment designer uses RL to generate complete, executable Python code representing Markov Decision Processes (MDPs), including state transitions, reward functions, and verification logic, unifying single-turn reasoning and multi-turn interactions within a single API. This code-based environment representation allows any computable environment to be expressed, moving beyond fixed, hand-crafted environments. The environment designer is trained via RL with a hint-based regret signal, which measures the gap in the reasoning agent’s performance with and without privileged hints. This encourages the environment generator to produce environments at the edge of the agent’s capabilities but still feasible. The reasoning agent interacts with these environments, learning through reinforcement signals, while both roles share parameters and are optimized jointly using Group Relative Policy Optimization (GRPO). Extensive experiments at scales up to 30B parameters demonstrate that SPADE outperforms fixed environment baselines by an average of 5.3% across eight benchmarks, with significant gains in tool-use tasks (+5.7% and +13.9%). The framework enables continual environment and agent co-evolution, supporting open-ended self-improvement.
Key Results
- Across eight diverse benchmarks—including math, science, coding, and reasoning tasks—SPADE achieves an average performance boost of 5.3% over the strongest fixed environment baselines. Notably, in tool-use scenarios such as BFCL-v4 multi-turn and ACEBench-Agent, performance improvements reach +5.7 and +13.9 points respectively. The results are consistent across model scales, with larger models exhibiting greater gains, indicating good scalability. The environments generated by SPADE evolve from simple tasks to complex, multi-constraint, multi-turn environments, demonstrating the system’s capacity for curriculum development and difficulty progression.
- Ablation studies reveal that the hint-based regret reward mechanism significantly outperforms traditional signals like exponential moving averages, confirming its effectiveness in guiding environment design. The use of a corpus-grounded, memory-augmented environment design pipeline enhances environment diversity and complexity, leading to better generalization. The generated environments incorporate knowledge from large pretraining corpora, enabling the model to create challenging yet solvable tasks grounded in real-world knowledge. Performance gains are especially pronounced in multi-turn reasoning and tool-use tasks, highlighting the framework’s strength in complex interactive scenarios.
- In game environments, performance improvements scale with model size, with larger models consistently outperforming baselines. The environments produced show a natural curriculum progression, starting from simple single-skill tasks and advancing to environments requiring long-horizon, multi-constraint reasoning. This emergent curriculum demonstrates SPADE’s ability to generate increasingly complex tasks, fostering continual learning and capability expansion. The results validate the framework’s potential for scalable, autonomous environment synthesis that adapts to the agent’s evolving skills.
Significance
This work addresses a fundamental bottleneck in AI development: the static nature of training environments. By enabling environments to be generated and optimized dynamically through a learnable process, SPADE paves the way for truly autonomous, self-improving AI systems. The approach reduces reliance on human-designed curricula and fixed synthetic environments, significantly lowering costs and increasing flexibility. It also introduces a new paradigm where environment complexity and diversity grow in tandem with the agent’s capabilities, fostering continuous learning and adaptation. This has profound implications for advancing artificial general intelligence (AGI), as it aligns with the vision of open-ended, lifelong learning systems capable of tackling increasingly complex tasks across diverse domains.
Technical Contribution
SPADE introduces a novel framework where environment design is integrated into the RL training loop as a learnable, dual-role process within a single large language model. Environments are represented as executable Python programs implementing full MDPs, enabling the generation of any computable environment rather than being limited to fixed, parameterized scenarios. The hint-based regret reward mechanism provides a principled way to steer environment generation towards the agent’s learning frontier, balancing challenge and feasibility. The use of GRPO ensures stable training at scale, and the incorporation of a corpus-grounded environment memory enhances diversity and complexity. These innovations collectively enable the co-evolution of environments and agents, supporting open-ended, continual self-improvement.
Novelty
The key novelty lies in transforming environment design from a static, human-crafted process into a dynamic, learnable component that co-evolves with the agent. Unlike prior works that generate tasks or fixed environments, SPADE synthesizes full multi-turn MDP environments as executable code, trained via RL with a hint-based regret signal grounded in performance metrics. This approach allows the environment generator to adaptively produce increasingly challenging environments aligned with the agent’s capabilities, facilitating continuous self-improvement. The integration of environment generation, validation, and optimization into a single framework at scale is a significant advancement over existing static or fixed synthetic environment methods.
Limitations
- Despite its scalability, SPADE requires substantial computational resources, especially for training large models (e.g., 30B parameters), which may limit practical deployment. The environment generation process depends heavily on the quality of pretraining corpora and environment memory, potentially restricting diversity if data coverage is insufficient. Additionally, the current framework primarily targets structured reasoning and tool-use tasks; extending it to unstructured or multimodal environments remains challenging. The complexity of environment validation and the risk of generating environments that are either too easy or too hard also pose ongoing challenges that need further refinement.
Future Work
Future research will focus on integrating multimodal inputs—such as visual and auditory data—to enrich environment representations and challenge agents more comprehensively. Improving environment validation efficiency and robustness is also a priority, aiming to reduce training costs and enhance scalability. Additionally, incorporating meta-learning and transfer learning techniques could enable environment generators to adapt across diverse domains more effectively. Long-term, the goal is to develop fully autonomous, lifelong learning systems capable of self-generating, self-validating, and self-improving environments, ultimately pushing AI towards general intelligence with minimal human intervention.
AI Executive Summary
In the rapidly evolving field of artificial intelligence, the capacity for continuous self-improvement remains a central challenge. Traditional approaches rely heavily on manually curated or static synthetic environments, which quickly become limiting as models scale and their capabilities expand. Recognizing this bottleneck, the SPADE framework introduces a groundbreaking approach that leverages a single large language model (LLM) to generate, optimize, and co-evolve training environments in tandem with the agent itself.
SPADE’s core innovation is to treat environment design as a learnable, dual-role process within the same model. The environment designer role involves generating complete, executable Python code representing Markov Decision Processes (MDPs), including state transitions, reward functions, and verification logic. These environments are flexible enough to encompass both single-turn reasoning tasks and complex multi-turn interactions, unified under a standard Gym-style API. This code-based representation allows the system to express any computable environment, moving beyond the limitations of fixed, hand-crafted scenarios.
The second role, as a reasoning agent, interacts with these environments, learning through reinforcement signals. The key to guiding environment design is a novel hint-based regret reward mechanism. By comparing the agent’s performance with and without privileged hints, the environment designer receives a signal that encourages it to produce environments at the edge of the agent’s current capabilities—challenging yet solvable. This dynamic creates a co-evolutionary loop where environments become progressively more complex as the agent improves, fostering continual learning.
Experimental results at scales up to 30 billion parameters demonstrate the effectiveness of SPADE. Across eight benchmarks spanning math, science, coding, and reasoning, the framework achieves an average performance boost of 5.3%, outperforming fixed environment baselines. In tool-use scenarios, improvements are even more pronounced, with gains of +5.7% and +13.9% on specific tasks. The generated environments exhibit a natural curriculum, starting from simple tasks and evolving into complex, multi-constraint environments requiring long-horizon reasoning.
This work represents a significant step toward autonomous, open-ended AI systems capable of self-generating and self-improving environments. By integrating environment design into the RL training process and enabling environments to grow in complexity alongside the agent, SPADE addresses longstanding limitations of static training regimes. Its scalable, adaptable approach holds promise for advancing artificial general intelligence, with potential applications in education, scientific research, robotics, and beyond. Future directions include multimodal environment synthesis, more efficient validation mechanisms, and broader domain adaptation, paving the way for truly autonomous lifelong learning systems.
Deep Dive
Abstract
Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.