SkillCAT: Contrastive, Assessment-Augmented and Topology-AwareSkill Self-Evolution for LLM Agents
SkillCAT framework boosts LLM skill self-evolution by 49.69% through Contrastive Causal Extraction, Assessment-Augmented Evolution, and Topology-Aware Task Execution.
Key Findings
Methodology
The SkillCAT framework consists of three stages: Contrastive Causal Extraction (CCE) samples multiple trajectories per task and contrasts same-task success/failure pairs to find evidence explaining outcome differences; Assessment-Augmented Evolution (AAE) replays each candidate patch on source-task clones, retains only those that do not damage task outcomes, and merges them hierarchically; Topology-Aware Task Execution (TTE) compiles evolved skills into routable sub-skill topologies, loading only task-relevant capability nodes.
Key Results
- SkillCAT improved the average score over initial skills by up to 49.69% on benchmarks like SpreadsheetBench, WikiTableQuestions, and DocVQA.
- In cross-model and out-of-distribution generalization assessments, SkillCAT demonstrated reliable and effective skill evolution.
- Each component (CCE, AAE, and TTE) contributed to the final improvement, validating the design's robustness.
Significance
SkillCAT optimizes the skill self-evolution process, enhancing LLM agents' performance in complex tasks by addressing single-trace bias, unvalidated merging, and inference-time context overload in existing methods.
Technical Contribution
SkillCAT introduces new skill self-evolution strategies through contrastive causal extraction and assessment-augmented evolution, significantly improving skill reliability and task-selective deployment capabilities.
Novelty
SkillCAT is the first to decompose the skill lifecycle into three observable stages, introducing targeted optimizations for each, significantly enhancing the efficiency and accuracy of skill self-evolution.
Limitations
- SkillCAT may experience performance drops when handling extremely complex tasks, as these may require more intricate skill topologies.
- Hierarchical merging of skills may lead to information loss in some cases.
Future Work
Future work could explore the application of SkillCAT to more task types and optimize topology-aware task execution to enhance handling of complex tasks.
AI Executive Summary
The SkillCAT framework aims to address key issues in existing LLM skill self-evolution methods. Current methods typically derive skill patches from a single trajectory per task and load the entire skill corpus during inference, leading to unreliable evidence extraction, accumulation of low-quality or harmful skill edits, and inefficient use of context due to irrelevant or conflicting skill content.
SkillCAT effectively addresses these issues by decomposing the skill self-evolution process into three stages: Contrastive Causal Extraction (CCE), Assessment-Augmented Evolution (AAE), and Topology-Aware Task Execution (TTE). CCE samples multiple trajectories per task and contrasts same-task success/failure pairs to find evidence explaining outcome differences; AAE replays each candidate patch on source-task clones, retains only those that do not damage task outcomes, and merges them hierarchically; TTE compiles evolved skills into routable sub-skill topologies, loading only task-relevant capability nodes.
Experimental results show that SkillCAT significantly improves the average score of skill self-evolution across multiple benchmarks, demonstrating its reliability and effectiveness. SkillCAT's design not only enhances skill reliability but also strengthens task-selective deployment capabilities, providing new directions for future research and applications.
Deep Dive
Abstract
Skill self-evolution methods for LLM agents aim to turn execution trajectories into reusable skill documents. However, current pipelines typically derive skill patches from a single trajectory per task, merge them indiscriminately, and load the entire skill corpus during inference. These choices lead to unreliable evidence extraction, the accumulation of low-quality or even harmful skill edits, and inefficient use of context due to irrelevant or conflicting skill content. We propose SkillCAT, a framework that decomposes this process into three stages. (1) Contrastive Causal Extraction (CCE) samples multiple trajectories per task and contrasts same-task success/failure pairs to find the evidence that explains outcome differences. (2) Assessment-Augmented Evolution (AAE) replays each candidate patch on source-task clones, retains only those that do not damage task outcomes, and then merges the retained patches hierarchically. (3) Topology-Aware Task Execution (TTE) compiles the evolved skills into routable sub-skill topologies, so that inference loads only task-relevant capability nodes. We evaluate SkillCAT on widely-used agent benchmarks, including SpreadsheetBench, WikiTableQuestions, and DocVQA, and further assess cross-model and out-of-distribution generalization. Across these settings, SkillCAT improves the average score over the initial skill by up to 49.69%, demonstrating reliable and effective skill evolution.