SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
SkillFlow benchmark demonstrates 8.43% improvement in task success via lifelong skill evolution, using a dual-agent iterative framework.
Key Findings
Methodology
This study introduces SkillFlow, a benchmark comprising 166 tasks across 20 families, structured via a Domain-Agnostic Execution Flow (DAEF). It employs a dual-agent iterative pipeline: one constructs task assets, the other reviews and refines tasks based on trajectory and rubric feedback. The protocol models agents starting with no skills, solving tasks sequentially, externalizing lessons through skill patches, and updating a skill library over time. Experiments with models like Claude Opus 4.6 show an 8.43-point increase in task success rate after lifelong skill evolution, highlighting the framework’s effectiveness.
Key Results
- Claude Opus 4.6’s task success rate increased from 62.65% to 71.08% after applying lifelong skill evolution, a significant improvement. Although skill usage was high at 66.87%, the actual utility gain was modest (+0.60 points for Kimi K2.5), and some models like Qwen-Coder-Next showed regression, indicating challenges in skill transfer and repair. The results reveal that stronger models can stabilize and refine skills over iterations, whereas weaker models struggle with cohesive skill integration.
- Model performance varied widely; models like Claude Opus 4.6 achieved stable improvements, while others faced issues like fragmented skills and error reinforcement. The experiments underscore that high skill usage does not necessarily translate into high utility, emphasizing the need for better skill representation and repair mechanisms. The analysis of failure modes suggests future research directions to enhance skill stability and transferability.
- Overall, the experiments demonstrate the potential of lifelong skill learning but also expose significant gaps, especially in models’ ability to maintain coherent, reusable skills over multiple tasks. The framework provides a systematic way to evaluate and improve continual learning capabilities, guiding future development of autonomous agents with robust, evolving skill sets.
Significance
This work advances the field of autonomous lifelong learning by providing a structured benchmark to evaluate skill discovery, repair, and transfer. It addresses the long-standing challenge of enabling models to autonomously build and maintain a coherent skill library, crucial for real-world applications such as robotics, automation, and intelligent decision-making. The ability to continually evolve skills reduces reliance on static knowledge bases, fostering more adaptable and resilient AI systems. The insights gained from this benchmark inform both theoretical understanding and practical engineering, paving the way for more autonomous, self-improving AI agents.
Technical Contribution
The paper introduces a novel task architecture based on DAEF, enabling systematic cross-task skill transfer. It combines a dual-agent iterative pipeline with explicit skill patching driven by trajectory and rubric feedback, integrating continual learning protocols. The approach emphasizes skill externalization, repair, and evolution, providing a comprehensive evaluation framework. The methodology bridges the gap between static skill repositories and dynamic, self-improving agents, offering new theoretical insights into skill stability and transferability. The experimental setup rigorously tests different models, revealing key failure modes and guiding future improvements.
Novelty
This is the first comprehensive benchmark integrating lifelong skill discovery, repair, and transfer within a unified, domain-agnostic framework. Unlike prior static skill repositories, SkillFlow emphasizes autonomous skill externalization and iterative refinement driven by real task experiences. The use of DAEF as a standardized workflow abstraction allows cross-domain applicability and systematic evaluation of skill evolution. This approach represents a significant departure from traditional static evaluation, pushing the frontier toward truly autonomous, self-improving agents.
Limitations
- Despite promising results, current models still face challenges in achieving stable, coherent skill integration, often exhibiting fragmentation and error reinforcement. The framework relies heavily on native model capabilities for skill patching, which limits generalization and robustness.
- The benchmark tasks, while diverse, are simulated and may not fully capture real-world complexities such as noisy data, unpredictable environments, or multi-modal inputs. Extending the framework to real-world scenarios remains an open challenge.
- Computational costs associated with iterative skill repair and evaluation are high, potentially limiting scalability. Future work should focus on improving efficiency, robustness, and the ability to handle more complex, real-world tasks.
Future Work
Future research will explore more sophisticated skill representations, leveraging meta-learning and reinforcement learning to enhance transferability and stability. Integrating external knowledge bases and reasoning modules could improve skill repair accuracy. Extending the benchmark to real-world robotic and industrial tasks will validate practical applicability. Additionally, developing more efficient algorithms for skill patching and evaluation will be crucial for scaling autonomous lifelong learning systems.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在学习一门新技能，比如弹钢琴。刚开始，你只会弹几首简单的曲子，遇到难题时会试着找出问题所在，然后练习改正。每次练习完，你都会记下经验，下次弹得更好。随着时间推移，你的琴技逐渐提高，能应对各种复杂的曲子。这就像机器人学习一样，它从一无所有开始，通过不断尝试、修正和总结，逐步建立起自己的“技能库”。每次失败，它会分析原因，修补技能，变得越来越厉害。这种持续学习和改进的过程，正是SkillFlow研究的核心思想。它让机器像你一样，自己发现问题、解决问题，不断变得更聪明。未来，这样的机器人可以自己学习新技能，解决更复杂的问题，就像你变成了钢琴高手一样。
Abstract
As the capability frontier of autonomous agents continues to expand, they are increasingly able to complete specialized tasks through plug-and-play external skills. Yet current benchmarks mostly test whether models can use provided skills, leaving open whether they can discover skills from experience, repair them after failure, and maintain a coherent library over time. We introduce SkillFlow, a benchmark of 166 tasks across 20 families in which task construction within each family follows a Domain-Agnostic Execution Flow (DAEF) that defines an agent workflow framework, allowing these tasks to share a consistent workflow. Agents are evaluated under an Agentic Lifelong Learning protocol in which they begin without skills, solve tasks sequentially within each family, externalize lessons through trajectory- and rubric-driven skill patches, and carry the updated library forward. Experiments reveal a substantial capability gap. For Claude Opus 4.6, lifelong skill evolution improves task success from 62.65% to 71.08% (+8.43 points). However, high skill usage does not necessarily imply high utility: Kimi K2.5 gains only +0.60 points despite 66.87% skill usage, while Qwen-Coder-Next reaches only a 44.58% task completion rate and still regresses relative to the vanilla setting. SkillFlow contributes a structured testbed for this direction and an in-depth empirical analysis of skill discovery, patching, transfer, and their failure modes under lifelong evaluation.