The PokeAgent Challenge: Competitive and Long-Context Learning at Scale

In the field of artificial intelligence, partial observability, game-theoretic reasoning, and long-horizon planning have been core challenges in sequential decision-making. However, existing benchmarks often focus on one aspect, such as imperfect-information games emphasizing equilibrium computation in short episodes, while open-ended environments test exploration but lack adversarial opponents. Pokemon is an environment that combines all three: competitive battles require reasoning under hidden information against a strategic adversary, while single-player campaigns demand thousands of cumulative decisions spanning exploration, resource management, and combat over extended horizons. Pokemon's complexity and dynamism make it a more complex testbed than most existing benchmarks. Recently, Pokemon has gained significant interest for evaluating frontier AI systems. Demonstrations like Claude Plays Pokemon, Gemini 2.5 Pro, and OpenAI's GPT-5 have reinforced Pokemon's suitability as an AI testbed, but efforts have been fragmented due to different games, harnesses, and evaluation criteria.

The PokeAgent Challenge aims to address the core challenges of partial observability, game-theoretic reasoning, and long-horizon planning. Existing benchmarks often focus on one aspect, whereas the PokeAgent Challenge simultaneously examines these three capabilities under realistic conditions through Pokemon's multi-agent battle system and RPG environment. Pokemon's complexity and dynamism make it a more complex testbed than most existing benchmarks. With a standardized evaluation framework and rich datasets, the PokeAgent Challenge offers an important research tool for academia and industry.

The core innovations of the PokeAgent Challenge lie in its standardized evaluation framework and rich datasets. Firstly, it combines competitive battling via Pokemon Showdown with RPG speedrunning via Pokemon Emerald, providing a living competitive ecosystem. Secondly, it offers the largest publicly available Pokemon battle dataset, comprising over 20 million battle trajectories. Lastly, it introduces the first open-source multi-agent orchestration system for long-horizon RPG play. These innovations enable the PokeAgent Challenge to simultaneously examine the capabilities of partial observability, game-theoretic reasoning, and long-horizon planning under realistic conditions.

The PokeAgent Challenge evaluates AI decision-making capabilities through two complementary tracks:

• �� Battling Track: Provides a dataset of over 20 million battle trajectories and includes heuristic, reinforcement learning (RL), and large language model (LLM)-based baselines.

• �� Speedrunning Track: Offers the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons.

• �� Dataset: The Battling Track dataset comprises over 20 million battle trajectories, while the Speedrunning Track dataset offers the first standardized evaluation framework for RPG speedrunning.

• �� Baselines: The Battling Track baselines include heuristic, RL, and LLM-based strategies, while the Speedrunning Track baselines offer the first standardized evaluation framework for RPG speedrunning.

The experimental design of the PokeAgent Challenge includes two complementary tracks: the Battling Track and the Speedrunning Track. The Battling Track provides a dataset of over 20 million battle trajectories and includes heuristic, reinforcement learning (RL), and large language model (LLM)-based baselines. The Speedrunning Track offers the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons. In the NeurIPS 2025 competition, over 100 teams participated across both tracks, revealing significant gaps between generalist LLM, specialist RL, and elite human performance.

The key results of the PokeAgent Challenge include:

• �� In the Battling Track, significant gaps were found between generalist LLM, specialist RL, and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites.

• �� The Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, with participants using novel methods such as Scripted Policy Distillation and iterative offline RL with dynamic data weighting.

• �� The NeurIPS 2025 competition attracted over 100 teams, revealing considerable gaps between generalist LLM, specialist RL, and elite human performance.

The application scenarios of the PokeAgent Challenge include:

• �� Academic Research: With a standardized evaluation framework and rich datasets, the PokeAgent Challenge offers an important research tool for academia.

• �� Industrial Applications: The PokeAgent Challenge provides a standardized platform for evaluating AI decision-making capabilities, particularly in partial observability, game-theoretic reasoning, and long-horizon planning.

• �� Game Development: The PokeAgent Challenge offers a standardized platform for testing AI decision-making capabilities, particularly in complex multi-agent systems and dynamic environments.

Despite the rich datasets and standardized evaluation framework, the complexity of the PokeAgent Challenge may lead to high computational costs, limiting participation by smaller research teams. Additionally, the complexity and dynamism of the Pokemon environment may pose challenges for models adapting to continuously evolving metagames. Future research directions include developing more efficient algorithms to address the challenge's complexity, particularly in partial observability and long-horizon planning.