EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments
EvoPolicyGym benchmarks autonomous policy evolution; GPT-5.5 achieves top performance across 16 RL environments with limited interactions.
Key Findings
Methodology
This study introduces EvoPolicyGym, a controlled benchmark where agents iteratively edit executable policies within a fixed interaction budget. The framework employs a Transformer-based GPT-5.5 model that interacts with a suite of 16 RL environments, including Gym/Box2D, MuJoCo, and robotics tasks. The process involves agents submitting policy revisions, receiving environment feedback—such as rollout summaries and diagnostics—and refining policies based on trajectory-level insights. The evaluation emphasizes not only final performance but also the dynamics of policy improvement, budget allocation, and mechanism discovery. The system integrates trajectory tracking, feedback utilization, and structural analysis to understand how agents optimize policies under resource constraints, highlighting the importance of discovering task-appropriate mechanisms and refining policies through bounded feedback.
Key Results
- GPT-5.5 achieved the highest aggregate rank score of 0.891 across all 16 environments, with nine first-place wins and 12 top-two placements. It consistently outperformed other models like Claude Opus 4.7, especially in Gym/Box2D, MuJoCo, and robotics tasks, demonstrating superior efficiency in policy refinement under limited interactions. The trajectory analysis revealed that successful policy evolution depends on effective feedback utilization and mechanism discovery, rather than isolated task wins. The model's ability to adapt structures and parameters within a constrained budget underscores its robustness and generalization capabilities.
- Across the benchmark suite, GPT-5.5 maintained stable high performance, with an average rank score of 0.891, covering diverse task families. Its strategies involved systematic exploration, targeted feedback conversion, and structural refinements, leading to consistent improvements. The experimental results highlight that trajectory-level diagnostics can reveal how agents allocate resources, diagnose failures, and discover task-specific mechanisms, providing deeper insights than final scores alone. The results validate the framework's effectiveness in evaluating iterative policy improvement under resource constraints.
- Trajectory analysis demonstrated that strong autonomous policy evolution hinges on the ability to discover task mechanisms and refine policies through bounded feedback. The models that excelled were those capable of structural synthesis and parametric tuning, translating sparse environment signals into robust policy updates. These findings suggest that future research should focus on enhancing feedback interpretation and mechanism discovery, especially in complex, real-world scenarios where resources are limited and tasks are diverse.
Significance
This work advances the evaluation paradigm for autonomous agents by shifting focus from static performance metrics to the process of policy evolution. The EvoPolicyGym benchmark provides a systematic platform to analyze how agents allocate limited resources, utilize environment feedback, and discover task-specific mechanisms. Such insights are crucial for developing autonomous systems capable of continuous learning and adaptation in resource-constrained settings. The framework bridges gaps between reinforcement learning, software engineering, and AI interpretability, offering a comprehensive tool for research and development. Its trajectory-level diagnostics enable researchers to understand the internal dynamics of policy improvement, fostering more transparent and robust autonomous agents. Overall, this study lays a foundation for future work on resource-efficient, self-improving AI systems that can operate reliably in complex, real-world environments.
Technical Contribution
The primary technical contribution is the development of EvoPolicyGym, a benchmark that formalizes autonomous policy evolution as an iterative, resource-constrained optimization process. It introduces a trajectory-level diagnostic system that captures how agents allocate budgets, convert feedback into parameter updates, and discover mechanisms. The framework employs a Transformer-based GPT-5.5 model integrated with a policy editing harness, enabling flexible, iterative policy refinement. The evaluation protocol enforces strict visibility boundaries, with feedback limited to training episodes and final evaluation based on hidden validation and held-out performance. This setup allows detailed analysis of policy evolution dynamics, including feedback utilization, structural synthesis, and exploration-exploitation trade-offs. The system's design facilitates cross-task comparison and provides insights into the internal decision-making processes of autonomous agents.
Novelty
This is the first comprehensive benchmark explicitly designed to evaluate autonomous policy evolution under bounded interaction budgets, emphasizing process transparency through trajectory analysis. Unlike prior work focused solely on final task performance or static code editing, EvoPolicyGym captures the iterative nature of policy refinement, integrating feedback-driven structural discovery. Its trajectory-level diagnostics and resource-aware evaluation protocol distinguish it from existing benchmarks, offering a new lens to study how autonomous agents discover mechanisms and optimize policies in resource-limited settings. This approach bridges reinforcement learning, software engineering, and interpretability, providing a novel framework for understanding continuous policy improvement.
Limitations
- The current benchmark relies on simulated environments, which may not fully capture the complexity and unpredictability of real-world scenarios. Generalization to physical systems remains to be validated.
- Limited interaction budgets constrain the exploration space, potentially hindering the discovery of more sophisticated mechanisms or long-term strategies.
- The framework emphasizes trajectory analysis but may not fully account for the computational costs associated with large models like GPT-5.5, which could limit scalability.
- Mechanism discovery remains challenging in highly complex tasks, and the current approach may require further enhancements to handle such scenarios effectively.
Future Work
Future research will extend EvoPolicyGym to real-world robotics and autonomous systems, validating its effectiveness beyond simulation. Efforts will focus on improving feedback interpretation, enabling more efficient exploration within limited budgets, and integrating multi-modal signals. Additionally, advancing the theoretical understanding of mechanism discovery and structural synthesis will be prioritized. Exploring multi-agent settings and transfer learning scenarios could further enhance the robustness and applicability of autonomous policy evolution. The ultimate goal is to develop autonomous agents capable of continuous, resource-efficient learning in dynamic, real-world environments, pushing the boundaries of AI autonomy and adaptability.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家厨房里做菜，你不断尝试不同的调料和烹饪方法，每次尝试后都根据味道调整。你手头的调料有限，不能无限制试验，但你希望做出最好吃的菜。这个研究就像是在设计一个聪明的厨师，它可以自己反复试验、学习，逐步改进菜谱。厨师每次尝试后会根据味道反馈调整调料比例或换用不同的调料。它还会记住哪些方法效果好，哪些不好，然后在有限的时间和调料中找到最优方案。科学家们用这个方法让机器人也能像厨师一样，学会在有限的资源下不断改进自己，变得越来越厉害。这个过程就像你在厨房里不断试错，最终做出最棒的菜。
Abstract
Autonomous agents are increasingly expected to improve executable policies through feedback, yet existing evaluations often collapse this process into a final score or confound it with open-ended software-engineering progress. We introduce Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget. We instantiate this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies. On the EvoPolicyGym suite, GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments. Beyond leaderboard results, EvoPolicyGym also provides trajectory-level diagnostics that distinguish how agents allocate budget, convert feedback into parametric tuning. These analyses show that strong autonomous policy evolution depends not only on isolated task wins, but on discovering task-appropriate mechanisms and refining policies under bounded feedback.