BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
BenchEvolver employs solution evolution to automatically generate harder, verifiable coding tasks, significantly reducing model success rates and enhancing discriminative evaluation.
Key Findings
Methodology
This paper introduces a solution-centric evolutionary framework that mutates reference solutions directly, deriving new task statements and tests from these evolved solutions. The process involves structured mutations, independent validation checks, and empirical difficulty assessment based on model failure rates. The framework includes components such as a proposer, evaluator, and memory modules, forming a closed-loop system that iteratively generates increasingly challenging tasks while ensuring correctness and diversity. Experiments on LiveCodeBench and SciCode demonstrate that evolved tasks substantially lower model pass rates, thereby improving benchmark discriminability and providing effective training signals for model self-improvement.
Key Results
- In LiveCodeBench-v6, the framework reduces Pass@1 success rates from over 99% to between 27.5% and 62.6%, significantly increasing task difficulty and model discriminability.
- In SciCode, evolved tasks also show marked difficulty increases, with validation confirming high correctness. Reinforcement learning on these tasks yields performance gains of +8.7 and +8.3 points for gpt-oss-20b, surpassing seed-only training by 70.7% and 34.8%.
- Compared to traditional synthetic methods, solution evolution ensures tasks are verifiable, diverse, and challenging for the generating model itself, fostering self-improvement.
Significance
This approach addresses the saturation of existing benchmarks, providing a scalable way to generate high-quality, challenging evaluation and training data. It enables models to expose and learn from their own weaknesses, facilitating continuous performance improvements. The method's ability to produce diverse, verifiable tasks across domains makes it a valuable tool for advancing AI capabilities in both research and industry, especially in scenarios requiring robust evaluation and self-improvement.
Technical Contribution
The key innovation is the direct evolution of executable solutions rather than just instructions, ensuring task validity and complexity. The framework integrates multi-round filtering, difficulty measurement via empirical failure rates, and memory-guided search to efficiently explore the solution space. This design allows automatic generation of high-quality, diverse, and increasingly difficult tasks, providing a new paradigm for self-challenging model evaluation and training.
Novelty
This work is the first to leverage solution-space evolution for automatic task generation, directly mutating reference solutions to produce verifiable, challenging tasks. Unlike prior methods focusing on instruction synthesis or static data, it ensures the generated tasks are logically consistent and difficult for the target models, enabling a self-improving cycle. This paradigm shift opens new avenues for scalable, autonomous benchmark creation.
Limitations
- The method relies on the target model's ability to accurately assess difficulty; biases or blind spots in the model may skew difficulty evaluation.
- Computational costs are high, especially for large models and complex tasks, limiting scalability without further optimization.
- In scientific domains, validation mechanisms need refinement to ensure scientific validity and relevance of generated tasks.
Future Work
Future directions include extending the framework to multi-modal tasks involving images or videos, improving efficiency via parallelization and better search strategies, and integrating human-in-the-loop validation for complex scientific tasks. Additionally, applying this approach to real-world industrial applications such as automated code review, adaptive tutoring systems, and autonomous agent training will be explored to realize its full potential.
AI Executive Summary
The rapid advancement of large language models has led to the saturation of standard evaluation benchmarks like LiveCodeBench, where top models achieve success rates exceeding 99% on easy tasks and over 90% on average across difficulty levels. This saturation diminishes the benchmarks’ ability to differentiate model capabilities, hindering progress in model development and assessment.
To address this challenge, the authors propose BenchEvolver, a novel solution-centric evolutionary framework that automatically transforms existing coding problems into more difficult, verifiable variants. Unlike traditional data augmentation or instruction-based synthesis, BenchEvolver directly mutates reference solutions, deriving new task statements and tests from these evolved solutions. This approach ensures that generated tasks are grounded in executable semantics, maintaining correctness while increasing difficulty.
The framework employs structured mutations, independent validation checks, and empirical difficulty measurement based on model failure rates. It incorporates memory modules to guide the search process, promoting diversity and avoiding repetitive solutions. Experiments on LiveCodeBench and SciCode demonstrate that the evolved tasks significantly reduce model success rates, thus restoring discriminative power among models. Moreover, reinforcement learning experiments show that training on these evolved tasks improves model performance substantially, surpassing seed-only training gains.
This work not only provides a scalable method for generating challenging benchmarks but also creates a self-improving loop where models can expose and learn from their weaknesses. The ability to generate diverse, high-quality tasks across domains promises to accelerate AI development, especially in areas requiring robust evaluation and continual learning. Future work will focus on extending the framework to multi-modal tasks, optimizing computational efficiency, and deploying in real-world scenarios such as automated coding and intelligent tutoring systems.
Deep Analysis
Background
近年来，随着GPT、Codex等大模型的崛起，代码理解与生成成为研究焦点。然而，随着模型性能不断突破，评测基准逐渐饱和，难以区分模型优劣。早期工作如CodeSearchNet、HumanEval提供基础评估，但难以应对模型持续提升带来的判别能力下降问题。研究者开始探索合成数据、对抗训练和演化算法，以自动生成更具挑战性的任务，但多依赖人工设计或指令级合成，缺乏对任务完整性和验证性的保障。本文提出方案空间演化策略，直接在解决方案层面操作，确保任务的可执行性和多样性，为持续评估和模型自我提升提供新工具。
Core Problem
现有基准的饱和严重限制了模型性能的提升空间，人工构建难题成本高且难以规模化。自动生成高难度、可验证任务面临验证机制不足、任务合理性难以保证等挑战。如何在保证任务正确性和多样性的同时，动态调节难度，成为核心问题。特别是在模型自我改进场景中，生成的任务必须能反映模型的真实弱点，否则难以推动性能提升。
Innovation
创新点在于方案空间的直接演化，利用参考解决方案进行变异，确保任务的逻辑严密和验证可行。引入多轮筛选机制，结合模型在目标任务上的表现，动态调节任务难度。区别于传统指令合成，方案演化保证了任务的可执行性和多样性。记忆机制的引入增强了搜索效率和多样性，形成闭环优化流程。这一策略突破了以往仅在指令层面合成任务的局限，为模型自我挑战提供了新途径。
Methodology
- �� 以参考解决方案为基础，进行结构化变异，生成新方案；
- �� 从变异方案中推导任务描述、测试用例，确保语义一致；
- �� 利用多轮筛选，结合模型在任务上的表现，筛除不难或无效任务；
- �� 引入记忆机制，记录成功与失败的变异路径，指导后续搜索；
- �� 通过验证机制确保任务的正确性和可执行性；
- �� 采用多模型评估，动态调节任务难度，形成闭环优化流程。
Experiments
在LiveCodeBench和SciCode两个领域中，采用不同难度级别的种子任务，利用多种演化模型（如gpt-4o、Gemini-3-Flash）进行演化。评估指标包括任务有效性、模型成功率变化和多样性。通过对比传统任务生成方法，验证演化任务的难度提升和判别能力增强。还结合强化学习，验证演化任务对模型性能提升的促进作用。实验中还进行了算法类别分析，显示演化任务涵盖更丰富的算法结构。
Results
演化任务显著降低模型Pass@1成功率，从99%降至27.5%-62.6%，增强了模型区分能力。RL训练在演化任务上提升了模型性能，达8.7点，超越纯种子训练70.7%。多样性分析显示，演化任务引入了更多复杂算法类别，拓宽了模型的挑战面。验证机制确保任务的有效性和正确性，演化过程稳定可靠。
Applications
该方法可广泛应用于自动代码评估、模型自我提升、智能教育和自动化测试等场景。通过自动生成高难度任务，帮助模型识别弱点，提升整体性能。未来还可结合多模态信息，扩展到科学计算、图像理解等领域，推动AI自主学习和持续改进。
Limitations & Outlook
目前依赖目标模型的评估能力，模型偏差可能影响难度判断。演化过程计算成本较高，尤其在大规模任务中资源消耗大。对某些复杂科学任务，验证机制仍需优化，确保科学合理性。未来需提升效率和验证机制的鲁棒性，拓展应用范围。
Plain Language Accessible to non-experts
想象你在玩一个拼图游戏，一开始的拼图比较简单，你很快就拼好了。可是你觉得还可以更难一些，于是你用自己拼好的拼图作为基础，试着用不同的拼法，把它变得更复杂。每次你都试着拼出一个更难的版本，然后测试一下自己能不能拼出来。这样一来，你的拼图就变得越来越难，自己也变得更厉害。这个过程就像让AI自己不断改进任务，让它变得更难解决，从而让模型变得更聪明。它不是从头开始设计新题，而是用已有的拼图，逐步变得更难，确保每个新题都合理、具有挑战性。这就像你在游戏中不断升级关卡，让自己变得更厉害。
ELI14 Explained like you're 14
想象你在玩一个拼图游戏，一开始的拼图很简单，你很快就拼好了。可是你觉得还可以更难一些，于是你用自己拼好的拼图作为基础，试着用不同的拼法，把它变得更复杂。每次你都试着拼出一个更难的版本，然后测试一下自己能不能拼出来。这样一来，你的拼图就变得越来越难，自己也变得更厉害。这个过程就像让AI自己设计更难的任务，让它不断挑战自己，变得更聪明。它不是从头开始设计新题，而是用自己已有的拼图，逐步变得更难，确保每个新题都合理、具有挑战性。这就像你在游戏中不断升级关卡，让自己变得更厉害。
Abstract
The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.