SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement
SkillAxe improves LLM skills by 28% through evaluation-guided self-refinement.
Key Findings
Methodology
SkillAxe is an unsupervised framework that diagnoses and refines LLM skills across four dimensions: quality impact, trigger precision, instruction compliance, and solution-path coverage. No labels or rewards needed.
Key Results
- SkillAxe improved LLM skills' pass rate by 28% on SkillsBench, closing 47-67% of the gap to human-authored skills.
- On SpreadsheetBench, SkillAxe raised the pass rate from 16.0% to 52.0% using only 22 skills.
- SkillAxe excels in multi-task environments, significantly enhancing skill coverage.
Significance
The study addresses the lack of diagnostic feedback in LLM skill development, providing an automated, unsupervised optimization method that significantly enhances skill effectiveness and reliability.
Technical Contribution
SkillAxe offers a novel approach to improve LLM skill quality and coverage through unsupervised diagnostics and refinement, surpassing existing manual tweaking methods.
Novelty
SkillAxe is the first to decompose skill quality into interpretable dimensions and optimize them using unsupervised methods, significantly enhancing skill effectiveness.
Limitations
- SkillAxe may face trigger precision issues in complex environments, requiring further optimization.
- The framework relies on the quality of existing skills, and low initial skill quality may affect final outcomes.
Future Work
Future work includes expanding SkillAxe to more domains, optimizing trigger precision, and developing more complex skill diagnostic mechanisms.
AI Executive Summary
SkillAxe is an innovative unsupervised framework designed to enhance the quality and coverage of LLM skills. Current LLM skill development lacks effective diagnostic feedback, making automated optimization difficult. SkillAxe diagnoses skills across four dimensions: quality impact, trigger precision, instruction compliance, and solution-path coverage. Experimental results show that SkillAxe increased pass rates by 28% on SkillsBench and raised pass rates from 16.0% to 52.0% on SpreadsheetBench. These results demonstrate SkillAxe's ability to significantly improve skill effectiveness and reliability, addressing long-standing pain points in LLM skill development. However, SkillAxe may face trigger precision issues in complex environments, and future work will focus on optimizing these aspects.
Deep Analysis
Background
LLM skill development is crucial for modern agent frameworks, but existing methods rely heavily on manual tweaking and lack systematic optimization mechanisms. SkillAxe addresses this issue through an unsupervised framework.
Core Problem
LLM skill development lacks effective diagnostic feedback, making optimization difficult. Existing methods cannot systematically identify the causes of skill failures.
Innovation
SkillAxe diagnoses skills across four dimensions and optimizes them using unsupervised methods, significantly enhancing skill effectiveness and coverage.
Methodology
- �� Quality Impact: Evaluates the overall effect of skills on outcomes
- �� Trigger Precision: Assesses skill activation on correct tasks
- �� Instruction Compliance: Evaluates agent adherence to skill guidance
- �� Solution-Path Coverage: Evaluates skill support for task execution strategies
Experiments
Experiments conducted on SkillsBench and SpreadsheetBench validate SkillAxe's effectiveness. Tests use multi-task environments and real datasets.
Results
SkillAxe significantly improves skill coverage and pass rates, excelling in multi-task environments and narrowing the gap to human-authored skills.
Applications
SkillAxe can be used for automated skill optimization, enhancing agent performance in complex tasks, applicable to multi-domain skill development.
Limitations & Outlook
SkillAxe may face trigger precision issues in complex environments, requiring further optimization. Low initial skill quality may affect final outcomes.
Plain Language Accessible to non-experts
Imagine SkillAxe as a chef who continuously improves their recipes. Each time they cook, they note what steps worked well and what needs improvement. This way, they constantly enhance their cooking skills without external guidance. SkillAxe is like this chef, using self-diagnosis and refinement to improve LLM skill quality.
ELI14 Explained like you're 14
SkillAxe is like a game character that upgrades its skills on its own. After completing a task, it analyzes which skills were effective and which need improvement. This way, it keeps getting better, just like you upgrade your character in a game!
Glossary
SkillAxe
An unsupervised framework for optimizing LLM skills through four diagnostic dimensions.
Used to improve the quality and coverage of LLM skills.
LLM
A model capable of generating natural language text, widely used in agent frameworks.
Serves as the core component for executing complex tasks.
SkillsBench
A benchmark environment for evaluating skill effectiveness, containing multiple tasks and skills.
Used to validate SkillAxe's effectiveness.
SpreadsheetBench
A benchmark environment for evaluating spreadsheet tasks, containing real-world tasks.
Used to validate SkillAxe's performance in complex tasks.
Unsupervised Learning
A machine learning method that does not require labeled data, improving model performance through self-optimization.
SkillAxe optimizes skills through unsupervised learning.
Open Questions Unanswered questions from this research
- 1 How to improve SkillAxe's trigger precision in complex environments?
- 2 What is SkillAxe's applicability across different domains?
- 3 How to optimize SkillAxe's initial skill quality?
Applications
Immediate Applications
Agent Skill Optimization
SkillAxe can be used for automated optimization of agent skills, increasing task completion rates. Applicable to multi-domain skill development.
Long-term Vision
Cross-domain Skill Development
SkillAxe can be expanded to more domains, improving agent performance in complex tasks. Trigger precision issues need to be resolved.
Abstract
Skill documents, structured natural-language instructions that guide Large Language Model (LLM) agents, are critical to modern agent frameworks, yet LLMs struggle to write skills that actually work. On SkillsBench, human-authored skills improve pass rates by 16.2 percentage points, while LLM-authored skills provide no measurable gain. We introduce SkillAxe, a fully unsupervised framework that enables LLMs to iteratively diagnose and refine their own skills. SkillAxe decomposes skill quality into four interpretable dimensions (quality impact, trigger precision, instruction compliance with fault attribution, and solution-path coverage), producing structured improvement briefs that require no ground-truth labels, test suites, or environment rewards. On SkillsBench, SkillAxe improves pass rates by 28\% relative over unimproved LLM skills and closes 47--67\% of the gap to human-authored skills. We validate the approach as a continuous improvement engine in the wild on SpreadsheetBench, where a SkillAxe-built skill library learns from past agent trajectories and raises pass rate from 16.0\% to 52.0\% using only 22 skills.