SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration
Flow-matching loss enables multi-strategy trajectory sampling, driving autonomous skill evolution.
Key Findings
Methodology
SkillFlow introduces Tempered Trajectory Balance (TTB), a flow-matching loss that samples trajectories proportionally to reward, preventing mode collapse and maintaining strategy diversity. It employs a trainable supervisor interacting with a structured environment with a dynamic skill library and a frozen executor. The framework jointly trains forward and backward policies, with the backward policy providing transparent per-step credit attribution at zero inference cost. Recursive skill evolution uses flow diagnostics—trajectory residuals, skill marginal flows, and step importance—to autonomously decide when, what, and where to update the skill library, enabling continuous autonomous growth. This integrated flow-based approach combines trajectory sampling, credit assignment, and skill management into a unified, scalable system.
Key Results
- On 14 datasets covering QA, math reasoning, code generation, and decision tasks, SkillFlow outperforms baselines, achieving 20-50% improvements. For example, WebShop accuracy increased from 54.10% to 87.96%, a 37.8% boost.
- In out-of-distribution tests, SkillFlow maintains strong performance, with an average F1 gain of about 6%, demonstrating excellent transferability. The strategy diversity and success rates surpass RL and static skill baselines.
- Ablation studies confirm TTB’s low variance advantage, and recursive skill evolution effectively expands the skill set, improving adaptability and robustness across tasks.
Significance
This work addresses fundamental challenges in reinforcement learning—strategy collapse, opaque credit attribution, and unprincipled skill evolution—by introducing a flow-based, reward-proportional sampling framework. It enables multi-path exploration, continuous skill growth, and transparent decision attribution, significantly advancing autonomous multi-task orchestration. The approach bridges theoretical innovation with practical scalability, offering a new paradigm for intelligent agents capable of self-evolving in complex environments, with broad implications for AI research and industry deployment.
Technical Contribution
The paper proposes a novel flow-matching loss (TTB) that aligns trajectory sampling with reward, ensuring strategy diversity. It integrates a backward policy for zero-cost credit attribution, and employs flow diagnostics for recursive skill management. The combined architecture enables stable, reward-proportional exploration and autonomous skill evolution without additional inference overhead, representing a significant departure from traditional RL methods relying on single-mode policies and heuristic skill updates.
Novelty
This is the first work to embed flow models into multi-path task orchestration with recursive skill evolution, leveraging reward-proportional trajectory sampling and zero-cost credit assignment. Unlike prior static or heuristic methods, SkillFlow dynamically manages skills and exploration paths, ensuring strategy diversity and transferability. Its integration of flow diagnostics with recursive skill management offers a new, scalable approach to autonomous agent development.
Limitations
- Training relies on extensive trajectory sampling, which can be computationally expensive, especially in large-scale environments.
- Skill library evolution may introduce redundancy or gaps in highly complex tasks, requiring further refinement.
- Sensitivity to environment structure and reward design necessitates careful hyperparameter tuning for optimal performance.
Future Work
Future directions include extending SkillFlow to multi-modal environments, integrating meta-learning for faster skill adaptation, and exploring multi-agent collaborations. Enhancing sampling efficiency and robustness, as well as applying the framework to real-world robotics and industrial automation, are promising avenues for advancing autonomous AI systems.
AI Executive Summary
The rapid progress of large language models (LLMs) has enabled the development of increasingly capable autonomous agents for complex tasks. However, existing task orchestration methods often rely on static skill sets, heuristic rules, or reward-driven reinforcement learning that suffers from strategy collapse and opaque credit assignment. These limitations hinder the agents’ ability to adapt, explore multiple strategies, and evolve their skills autonomously.
SkillFlow introduces a flow-based framework that fundamentally rethinks task orchestration by leveraging a flow-matching loss called Tempered Trajectory Balance (TTB). This loss ensures that the sampling probability of each trajectory is proportional to its reward, maintaining multiple high-reward paths and preventing the collapse into a single optimal strategy. The framework employs a trainable supervisor that interacts with a structured environment containing a dynamic skill library and a frozen executor. During multi-turn interactions, the supervisor generates reasoning, actions, and parameters, while the flow models jointly train forward and backward policies. The backward policy provides transparent, per-step credit attribution at zero additional inference cost, enabling precise understanding of decision contributions.
Building on these diagnostics, SkillFlow implements a recursive skill evolution mechanism. It uses flow residuals, step importance, and skill marginal flows to decide when to create, prune, or refine skills, effectively closing the loop from training signals to autonomous capability growth. Extensive experiments across 14 datasets demonstrate that SkillFlow surpasses state-of-the-art baselines, including direct LLMs, RL algorithms, and static skill methods. It achieves significant improvements in accuracy, strategy diversity, and efficiency, with gains of up to 50% in some tasks.
The broader impact of this work lies in its ability to enable autonomous systems that can self-evolve, adapt to new environments, and handle complex multi-task scenarios. It offers a scalable, theoretically grounded approach to continuous skill development, paving the way for more intelligent, flexible, and reliable AI agents. Despite its success, challenges remain in reducing training costs, improving skill library management, and extending applications to real-world settings, which will be the focus of future research.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家厨房里做饭。每次做菜都需要不同的步骤，比如切菜、炒菜、调味。以前的方法就像提前准备好所有菜谱，流程固定，遇到新菜就很难调整。SkillFlow就像一个聪明的厨师，它能根据食材和味道的反馈，自己调整做菜的步骤，甚至发明新菜。它用一种特殊的“流”机制，确保每个步骤都能得到奖励（比如味道好），而不是只追求最快或最便宜。这样，厨师可以不断学习，做出更好吃的菜，而且还能根据不同的食材和客人偏好，灵活变换菜谱。这个系统还能自己决定什么时候加入新菜，什么时候删掉不受欢迎的菜，变得越来越聪明。就像一个会自己学习、不断改进的厨师，能应对各种复杂的厨房挑战。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你要完成很多任务，比如解谜、打怪、收集宝藏。以前的游戏设计师会给你一套固定的攻略，告诉你怎么做，但如果遇到新怪或者新任务，就不知道怎么办了。SkillFlow就像一个聪明的朋友，它能自己学习怎么应对不同的挑战。它会观察你每一步的成功和失败，然后自己总结出哪些策略最有效。它用一种特别的方法，确保它不会只学会一种套路，而是学会很多不同的办法。这样，无论游戏怎么变，它都能找到应对的方法。它还能自己决定什么时候学习新技能，什么时候放弃一些不再需要的技能。就像一个不断变强、越来越聪明的伙伴，帮你在游戏中取得胜利。这个系统让AI变得更聪明、更灵活，能应对各种复杂的任务和环境，就像你在游戏中不断成长一样。
Abstract
In recent years, a variety of powerful LLM-based agentic systems have been applied to automate complex tasks through task orchestration. However, existing orchestration methods still face key challenges, including strategy collapse under reward maximization, high gradient variance with opaque credit assignment, and unguided skill evolution whose decisions are typically made by directly prompting an LLM to judge rather than derived from principled training signals. To address these challenges, we propose SkillFlow, a flow-based framework that takes a trainable Supervisor as the agent and a structured environment with dynamic skill library and frozen executor, automating task orchestration through multi-turn interaction. SkillFlow employs Tempered Trajectory Balance (TTB), a regression-based flow-matching loss that samples trajectories proportional to reward, preserving diverse orchestration strategies rather than collapsing to a single mode. The same flow objective yields a jointly learned backward policy that provides transparent per-step credit assignment at zero additional inference cost. Building on these flow diagnostics, a recursive skill evolution mechanism determines when to evolve, what skills to create or prune, and where decision gaps lie -- closing the loop from training signal to autonomous capability growth. Experimental results on 14 datasets show that SkillFlow significantly outperforms baselines across question answering, mathematical reasoning, code generation, and real-world interactive decision making tasks. Our code is available at https://anonymous.4open.science/r/SkillFlow-E850.