Better, Faster, Stronger: Programmatic Skill Learning Best Reduces Agent Cost
Proposes SpeedRunner, a programmatic skill learning method that reduces agent costs by analyzing trajectories without replay, achieving over 60% cost savings and performance gains.
Key Findings
Methodology
This paper introduces SpeedRunner, a framework that employs trajectory analysis via code execution environments to automatically extract and refactor skills. It operates in a two-phase cycle: during wake, an LLM generates trajectories; during sleep, a code agent analyzes these trajectories, diagnosing failures, identifying patterns, and updating a hierarchical skill library. The core algorithm involves constructing call graphs, detecting recurring behaviors, and refining skills without relying on replay buffers or validation. The approach leverages rich signals from errors, success metrics, and behavioral patterns embedded in trajectories, enabling continuous, online skill learning even in long-horizon tasks. This method overcomes the context window limitations of traditional methods, supporting scalable, structured skill induction.
Key Results
- Across three diverse environments—ScienceWorld, Crafter, and BabyAI—SpeedRunner consistently outperforms baselines in both performance and cost metrics. It reduces average token cost by over 60%, while achieving near-perfect success rates in BabyAI (from 67% to nearly 100%) and significant improvements in task completion in other environments.
- The experiments demonstrate that programmatic skills facilitate effective reuse across episodes, leading to a substantial decrease in repetitive primitive actions. Compared to ASI and OPO, SpeedRunner produces deeper, more hierarchical call graphs, indicating more structured and reusable skills.
- Ablation studies confirm that the code interpreter and trajectory analysis are critical components; removing them results in performance drops and increased costs, validating the importance of structured programmatic analysis for cost-effective skill learning.
Significance
This work addresses fundamental limitations of existing skill learning approaches that rely on natural language prompts or environment replay, which are often costly and brittle. By enabling online, continuous, programmatic skill induction solely from trajectory data, it offers a scalable solution for deploying autonomous agents in real-world, dynamic environments. The approach enhances robustness against environmental randomness and distribution shifts, paving the way for more adaptable, cost-efficient AI systems in robotics, industrial automation, and beyond. Its ability to learn structured hierarchies of skills from experience marks a significant step forward in lifelong learning for autonomous agents.
Technical Contribution
The key technical innovation lies in transforming trajectory data into executable call graphs, enabling hierarchical skill organization and reuse. The integration of a code interpreter allows dynamic analysis and editing of skills, supporting incremental improvements without environment resets. The dual-phase learning cycle—generating trajectories and analyzing them with code—facilitates online, continual skill refinement. This approach departs from prior methods that depend on prompt tuning or replay-based validation, offering a more scalable, structured, and cost-effective paradigm for skill acquisition. Extensive experiments across multiple environments demonstrate its generality and efficiency, establishing new benchmarks for online skill learning.
Novelty
This study is the first to leverage trajectory analysis for direct, online programmatic skill induction without environment replay or validation. Unlike prior work relying on natural language prompts or static code libraries, SpeedRunner dynamically refactors skills based on rich signals embedded in trajectories. Its hierarchical call graph structure captures reusable abstractions, enabling scalable, long-horizon task solving. This combination of trajectory-driven analysis, code-based skill representation, and hierarchical organization constitutes a novel paradigm in lifelong skill learning for autonomous agents.
Limitations
- The method's reliance on rich trajectory signals may limit robustness in environments with sparse feedback or high noise levels, where errors and success signals are less informative.
- Deterministic, programmatic skills can be brittle under high environmental randomness, requiring further mechanisms for stochasticity or robustness.
- Computational overhead from call graph analysis and code refactoring may increase with skill library complexity, necessitating optimization for real-time deployment.
Future Work
Future directions include integrating multi-modal signals (visual, tactile) to enhance signal richness, developing adaptive call graph structures for better scalability, and extending to multi-agent settings with collaborative skill sharing. Additionally, exploring probabilistic or stochastic programmatic skills could improve robustness in noisy environments. Long-term, this framework aims to facilitate autonomous, lifelong learning in real-world robotics and industrial systems, enabling agents to continuously evolve their skill sets with minimal human intervention.
AI Executive Summary
The rapid advancement of large language models (LLMs) has spurred interest in autonomous skill acquisition for intelligent agents operating in complex environments. Traditional approaches, such as prompt tuning or replay-based reinforcement learning, often incur high costs and face limitations in long-horizon, dynamic tasks. Addressing these challenges, this paper introduces SpeedRunner, a novel framework that leverages trajectory analysis via code execution environments to induce programmatic skills online, without relying on environment resets or replay buffers.
SpeedRunner operates through a two-phase cycle: in the wake phase, an LLM-based actor generates task trajectories; in the sleep phase, a code-based inducer analyzes these trajectories, diagnosing failures, detecting recurring behavior patterns, and refactoring skills into hierarchical call graphs. This process enables the agent to learn reusable, structured skills incrementally, significantly reducing exploration costs while maintaining or improving task performance. The core innovation lies in transforming raw trajectory data into executable, hierarchical call graphs that capture complex behaviors, enabling efficient skill reuse and long-term adaptation.
Experimental results across three diverse environments—ScienceWorld, Crafter, and BabyAI—demonstrate that SpeedRunner consistently outperforms baseline methods such as ReAct, OPO, and ASI. It achieves over 60% reduction in token costs, while boosting success rates, e.g., from 67% to near 100% in BabyAI. Ablation studies confirm that the code interpreter and trajectory analysis are critical for these gains, highlighting the importance of structured programmatic analysis. The approach also exhibits robustness to environmental randomness and distribution shifts, making it promising for real-world deployment.
This work marks a significant step toward scalable, cost-efficient lifelong learning in autonomous agents. By enabling continuous, online skill induction solely from experience, it opens new avenues for robotics, industrial automation, and intelligent systems, fostering adaptable and resilient AI capable of operating in unpredictable environments. Future research will focus on integrating multi-modal signals, optimizing call graph analysis, and extending to multi-agent collaboration, aiming to realize truly autonomous, lifelong learning agents.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做饭。以前，每次做菜都得按照固定的步骤，从头到尾一遍遍试，浪费时间也容易出错。现在，有了程序化技能，就像你提前写好一份详细的食谱，遇到类似的菜肴时，只要照着菜单操作就行了。厨师还会回看自己之前做菜的视频，从中总结出哪些步骤可以重复用，哪些需要改进。这样一来，不仅做菜快了，还能做得更好吃。机器人也是一样，它们通过分析之前的“做菜录像”，学会用代码把重复的动作变成“技能”，以后遇到类似任务就能快速调用，省时省力。这种方法让机器人变得更聪明、更省钱，也更能适应复杂环境，就像你变成了厨房里的高手一样！
ELI14 Explained like you're 14
想象你在玩一款超级复杂的游戏，每次都要从头开始试错，失败了还得重新来，非常浪费时间。现在，如果你能记住自己成功的操作步骤，把它写成攻略，下次遇到类似情况就可以直接用这个攻略，节省很多时间。这就是程序化技能的核心思想。研究人员用电脑分析机器人之前的操作轨迹，找出反复出现的行为和错误，然后用代码把这些行为变成“技能”，让机器人以后遇到类似任务时，直接调用这些技能，而不用每次都重新试错。这就像你用攻略打游戏，不仅快，还能变得更厉害。这个方法让机器人变得更聪明、更省钱，也更能适应复杂的环境，就像你变成了游戏高手一样！
Abstract
Recently, the practice of augmenting LLM agent capability with skills has gained prevalence. We explore the cost effective adaptation of agents to novel domains by means of learning skills. Existing works focus on performance gain over cost effectiveness. As a result, little is known about what skill learning strategies save cost. We argue that among all the different skill learning methods, those that view skills as programs can achieve the best cost reduction. By executing sequences of actions deterministically, a program-augmented agent can reliably and cheaply achieve goals that would otherwise require trial and error and risk degenerate behavior over long horizons. An agent can learn at inference time by incrementally discovering these programs and equipping them for future tasks. We hypothesize that past trajectories contain enough signal to guide skill learning, even without replay or validation, provided the agent can learn to analyze them. To test our claims, we propose SpeedRunner, a coding agent that analyzes trajectories and refactors skills for better performance on future tasks. Across three different embodied environments, we show that SpeedRunner consistently achieves the frontier in learning and cost reduction while remaining robust against distribution shifts and environmental randomness.