Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
MetaEvolve uses reinforcement learning to cultivate self-evolution meta-skills, boosting code task performance by over 10%.
Key Findings
Methodology
MetaEvolve synthesizes evolution-aware data, employs environment feedback via program execution for continuous rewards, and trains models with Group Relative Policy Optimization (GRPO). It constructs multi-round evolution trajectories from large-scale code datasets, enabling models to learn reflection, feedback utilization, and iterative refinement. The training process emphasizes goal-oriented improvement, fostering transferable meta-skills across domains. Experiments across seven benchmarks show significant improvements: 10.01% in-distribution, 24.12% out-of-distribution, and 46.9% in open-ended optimization tasks.
Key Results
- MetaEvolve outperforms baselines by 10.01% (in-distribution) and 24.12% (out-of-distribution) on seven coding benchmarks, demonstrating strong generalization.
- On open-ended algorithm optimization, it achieves a 46.9% relative improvement, indicating effective transfer of meta-skills.
- Analysis reveals more diverse, innovative solutions with goal-directed self-reflection behaviors, surpassing AlphaEvolve in structural diversity and novelty.
Significance
This work advances AI towards autonomous self-evolution by explicitly training models to reflect, learn from feedback, and improve over multiple rounds. It addresses the limitations of traditional single-pass training, enabling models to adapt and optimize in complex, open environments. The approach bridges the gap between static training and dynamic inference, offering a scalable pathway for developing more capable, adaptable AI systems that can generalize beyond narrow tasks, thus impacting both academia and industry.
Technical Contribution
Introducing a novel training paradigm that combines data synthesis of evolution trajectories, environment feedback via program execution, and reinforcement learning (GRPO). This framework explicitly cultivates core meta-skills like self-reflection and multi-round optimization, enabling models to learn goal-directed improvement strategies. The approach is scalable, domain-agnostic, and demonstrates superior transferability, representing a significant step beyond existing methods focused solely on task-specific fine-tuning or single-turn inference.
Novelty
This is the first systematic integration of reinforcement learning with evolutionary search to explicitly train large language models for self-evolution. Unlike prior work limited to post-training heuristics, MetaEvolve embeds multi-round reflection and feedback-driven refinement into the training process, emphasizing continuous improvement signals derived from program execution. Its ability to generate diverse, innovative solutions and transfer meta-skills across domains marks a key innovation.
Limitations
- The reliance on program execution environments limits applicability to tasks where such feedback is available. Extending to non-coding domains remains challenging.
- Training involves high computational costs due to large-scale data synthesis and multi-round evolution, which may hinder scalability.
- Model's reflection and self-improvement in highly complex or novel tasks are still limited, requiring further research on enhancing generalization and efficiency.
Future Work
Future directions include expanding the framework to other domains like natural language understanding and scientific computing, integrating self-supervised learning for better generalization, and optimizing training efficiency. Exploring more sophisticated feedback mechanisms and meta-skill architectures will further enhance autonomous self-evolution capabilities.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized many AI applications, yet their ability to autonomously improve and adapt remains limited. Traditional training paradigms focus on task-specific fine-tuning, leaving models passive during inference. Recent frameworks like AlphaEvolve demonstrated the potential of iterative self-evolution driven by environment feedback, but lacked a systematic training approach to cultivate the underlying meta-skills necessary for sustained self-improvement.
This paper introduces MetaEvolve, a novel framework designed to explicitly train LLMs to develop core meta-skills such as self-reflection, feedback utilization, and multi-round refinement. The key innovation lies in synthesizing evolution-aware data from large-scale code datasets, where program execution provides continuous reward signals based on correctness and efficiency. These signals are used to construct multi-round evolution trajectories, which serve as training samples. The model is trained via Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm that incentivizes progressive solution improvement.
By grounding the training process in coding tasks, MetaEvolve leverages the natural, fine-grained reward signals from program execution, enabling the model to learn strategies for diagnosing weaknesses, learning from prior attempts, and translating feedback into meaningful improvements. Extensive experiments across seven coding benchmarks show that MetaEvolve outperforms strong baselines, achieving over 10% absolute gains in in-distribution tasks and more than 24% in out-of-distribution tasks. Notably, in open-ended algorithm optimization problems outside the training domain, it attains a 46.9% relative improvement, demonstrating robust transferability.
Further analysis reveals that MetaEvolve produces solutions with higher structural diversity and novelty, indicative of genuine exploration rather than superficial patching. The learned meta-skills enable the model to generate more innovative, effective solutions across diverse tasks, including scientific computing and algorithm design. These results suggest that explicitly cultivating self-evolution capabilities is a promising pathway toward autonomous, self-improving AI.
Future work will explore extending this approach to broader domains, improving training efficiency, and enhancing the generalization of meta-skills. Overall, MetaEvolve marks a significant step toward AI systems capable of continuous self-improvement, reducing reliance on human intervention, and adapting to complex, open-ended challenges.
Deep Analysis
Background
近年来，随着GPT系列、Codex等模型的出现，LLMs在自然语言理解和代码生成中取得巨大突破。强化学习（RL）和演化算法在优化策略和探索空间方面发挥重要作用，但多集中于微调或单轮优化。AlphaEvolve等方法引入环境反馈，展示了多轮自我改进的潜力，但缺乏系统性训练机制，难以培养模型的核心元技能。传统方法多依赖人工设计或有限反馈，难以实现大规模、泛化的自我演化。推动模型具备多轮反思和策略调整能力，成为AI自主学习的重要方向。
Core Problem
核心问题在于如何系统性培养模型的反思、多轮改进能力，以实现自主演化。现有微调方法多为单次优化，难以模拟推理中的多轮反馈过程，模型在面对新任务时缺乏有效的策略调整能力。缺少连续奖励信号和演化轨迹的训练机制，限制了模型的自主学习潜力。如何利用程序执行提供的连续奖励，合成多轮演化轨迹，训练模型实现目标导向的反思与改进，是亟待解决的难题。
Innovation
本研究的创新在于：1）利用程序执行提供连续奖励信号，结合正确性与效率指标，构建丰富的训练样本；2）通过数据合成模拟推理时的多轮演化轨迹，训练模型学习反思和策略调整；3）引入GRPO强化学习算法，指导模型逐步优化方案。该框架突破了传统单轮优化的限制，强调模型在训练中学习目标导向的自我反思，促进迁移到未知任务。创新点在于将演化搜索与RL深度结合，系统性培养模型的自我演化能力。
Methodology
- �� 构建多源编码数据集（如APPS、TACO、CodeContests、Codeforces），采样多样响应，筛选高质量、多样性样本。
- �� 设计响应合成机制，模拟推理中的多轮演化轨迹，包括当前程序、性能评分和历史尝试。
- �� 利用程序执行提供连续奖励信号，结合正确性和运行时间，构建训练目标。
- �� 采用Group Relative Policy Optimization（GRPO）强化学习算法，优化模型策略，鼓励逐步改进。
- �� 在大规模代码数据上训练模型，强化反思和多轮优化能力。
- �� 通过多任务、多域评估验证模型迁移和泛化能力。
Experiments
采用7个编码基准（APPS、TACO、CodeContests、Codeforces、Atcoder、Leetcode、USACO）进行评估，比较MetaEvolve与AlphaEvolve等基线。训练中调优超参数（演化轮数、样本数、保留比例），测量准确率、运行时间和方案多样性。还进行ablation研究，分析不同超参数对性能的影响。结果显示，MetaEvolve在多项指标上显著优于对比方法，验证其有效性。
Results
MetaEvolve在7个编码任务中，平均提升10.01%（分布内）和24.12%（分布外），算法优化任务中达成46.9%的相对改进。生成的方案更具结构多样性和创新性，表现出目标导向的自我反思行为。模型在多轮演化中逐步改善性能，策略调整明显优于传统方法，展现出强泛化能力。
Applications
该方法适用于自动代码优化、算法设计、自动调试等场景，尤其在缺乏丰富标注或环境反馈时，能自主学习改进策略。未来可扩展到自然语言推理、科学计算等领域，推动自主学习系统的普及。
Limitations & Outlook
目前方法依赖程序执行环境，适用范围有限，需扩展到非编码任务。训练成本较高，尤其在大规模数据和多轮演化中，计算资源消耗大。模型在极端复杂或新颖任务中的反思能力仍有限，需增强泛化和效率。
Plain Language Accessible to non-experts
想象你在厨房做菜，刚开始做的菜可能还不够好。你尝尝味道，发现太咸或太淡，然后根据反馈调整配料。每次尝试后，你都在学习怎么做得更好。这就像模型在学习做菜，不断反思和调整，逐步变得更棒。MetaEvolve就像教会AI这个“厨师”如何自己反思、改进菜肴，通过不断试错，最终能做出既快又好吃的菜。它不是简单重复做一样的菜，而是学会了怎么改进每次做法，变得越来越厉害。这样，AI就能在没有人一直指导的情况下，自己变得更聪明、更能解决新问题，就像一个不断学习、变得更好的厨师一样。
ELI14 Explained like you're 14
想象你在玩一个游戏，刚开始你会输很多次，但每次你都总结经验，知道哪里出错，然后试着改正。慢慢地，你变得越来越厉害，能赢得更多比赛。MetaEvolve就像教AI这个“游戏高手”如何自己反思和改进，每次遇到难题，它都能分析哪里出错，尝试不同的办法，逐步变得更聪明。它通过不断练习和总结，学会了在新挑战中也能找到解决办法。就像你通过不断玩游戏变得更厉害一样，AI也在不断学习，变得更自主、更聪明。这种能力让它可以自己解决很多复杂的问题，不需要人一直教它怎么做。
Glossary
Meta-Skills (元技能)
指模型在解决问题时展现的反思、学习和自我改进能力，是超越单纯任务技能的高级能力。
论文中强调通过强化学习培养模型的自我反思和多轮优化能力。
Reinforcement Learning (强化学习)
一种机器学习方法，通过奖励信号引导模型逐步优化策略，强调试错和反馈利用。
MetaEvolve利用强化学习训练模型，奖励来自程序执行的连续信号。
Evolutionary Search (演化搜索)
模拟自然选择的优化算法，通过生成、评估和选择候选方案实现问题的逐步改进。
用在AlphaEvolve和MetaEvolve中，指导代码的多轮优化。
Program Execution (程序执行)
运行代码以验证其正确性和效率，提供连续的性能反馈信号。
作为奖励信号，驱动模型在编码任务中的自我改进。
Open Questions Unanswered questions from this research
- 1 如何将此方法推广到自然语言理解或科学计算等非编码领域，仍缺乏系统性验证。
- 2 模型在极端复杂或未知任务中的反思能力和泛化能力仍待提升。
Applications
Immediate Applications
自动代码优化
利用MetaEvolve自动改进程序性能，减少人工调试时间，适用于软件开发和算法优化。
智能调试系统
构建自主调试工具，自动识别和修复代码中的缺陷，提高软件质量。
Long-term Vision
自主学习AI系统
发展具备多轮自我反思和改进能力的AI，能在多领域自主学习和创新，推动智能系统的普及。
Abstract
Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains. We hypothesize that the success of such evolution frameworks hinges on meta-skills, such as self-reflection with environment feedback, that enable effective multi-round refinement, yet are largely neglected by traditional post-training. To bridge this gap, we present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning (RL), and inference-time evolutionary search. Concretely, we ground MetaEvolve in coding, where program execution provides natural, continuous reward signals beyond binary correctness. Building on these signals, we synthesize evolution trajectories as training data, each containing a current program, its fitness score (combining correctness and efficiency), and a history of prior attempts, and train the model via RL with verifiable rewards derived from test case execution. By training on large-scale code data, we aim to inspire generalizable domain-agnostic meta-skills that can transfer broadly to open-ended problems where such rich training signals are scarce. Across seven coding benchmarks, MetaEvolve outperforms the strongest baseline by 10.01% absolute on in-distribution tasks and 24.12% on out-of-distribution tasks. On open-ended algorithm optimization problems entirely outside the training domain, it further achieves a 46.9% relative improvement. These results demonstrate that explicitly cultivating self-evolution meta-skills offers a principled path toward more capable and autonomously self-evolving AI.