Skill Coverage: A Test Adequacy Metric for Agent Skills
Proposes skill coverage metric based on trajectory detection of skill behavior constraints, achieving 38.66%-45.51% coverage; improves failed task recovery by 16%.
Key Findings
Methodology
This paper introduces a trajectory-based skill coverage metric, transforming natural language skill instructions into semi-structured Skill Behavior Constraints (SBC). Using LLM-assisted extraction and validation, the approach assesses whether agent trajectories instantiate these SBCs by checking applicability conditions and expected behaviors. The validation against human annotations achieves 88.58% agreement, ensuring reliability. Experiments on SkillsBench reveal that current trajectories cover only 38.66%-45.51% of SBCs, indicating significant under-testing of skill behaviors. The framework uses failed SBCs as diagnostic targets, emphasizing original instructions to strengthen skills, which leads to an average 16% recovery in failed tasks.
Key Results
- Analysis shows that only 38.66%-45.51% of skill behavior constraints are covered by trajectories in SkillsBench, highlighting a gap between task success and skill-level verification.
- Failed constraint labels identify specific skill violations; reinforcing these instructions improves task success rate by 16%.
- Validation against human judgments yields 88.58% agreement, confirming the robustness of the automated detection protocol.
Significance
This work addresses a critical gap in evaluating reusable agent skills by providing a fine-grained, quantifiable measure of skill usage. It enables developers to diagnose skill deficiencies precisely, facilitating targeted improvements. The method enhances the interpretability and robustness of LLM-based agents, supporting scalable skill testing and iterative refinement. Such detailed assessment paves the way for more reliable, explainable AI systems capable of complex, multi-step tasks in real-world applications.
Technical Contribution
The paper introduces a formal SBC model that translates natural language instructions into verifiable constraints, combined with a validated trajectory detection protocol. The approach separates coverage assessment from task success, enabling precise diagnostics. The integration of human validation ensures high reliability, and the failure-guided strengthening mechanism provides a systematic way to improve skills iteratively, surpassing traditional success metrics.
Novelty
This is the first systematic framework to quantify skill coverage via trajectory detection of behavior constraints derived from natural language instructions. It bridges the gap between high-level task success and low-level skill execution, offering a scalable, fine-grained evaluation method that can be integrated into existing benchmarks and development workflows.
Limitations
- The method relies on observable trajectory data; some skill behaviors may not manifest in test runs, leading to underestimation of coverage.
- Natural language to constraint conversion may introduce ambiguity, affecting detection accuracy.
- Current focus on static instructions; extending to multi-modal, dynamic skills remains future work, requiring richer data sources.
Future Work
Future directions include integrating multi-modal signals (visual, auditory), automating constraint generation and repair, and scaling the approach to broader skill ecosystems. Developing adaptive, context-aware SBC models and embedding this framework into continuous learning pipelines will further enhance the robustness and applicability of skill assessment in real-world AI systems.
AI Executive Summary
The rapid advancement of large language models (LLMs) has led to the emergence of modular, reusable skills that guide AI agents across diverse tasks. However, current evaluation metrics primarily focus on task-level success, leaving a significant blind spot regarding the internal use and effectiveness of these skills. This gap hampers systematic improvement and interpretability of AI behaviors. Addressing this challenge, the paper introduces a novel skill coverage metric based on trajectory analysis of skill behavior constraints (SBC). SBCs are semi-structured representations derived from natural language instructions, specifying when and how an agent should behave under certain conditions.
The core innovation lies in leveraging large language models to extract candidate SBCs from skill documents, followed by human validation to ensure accuracy. The framework then assesses whether agent trajectories instantiate these SBCs by checking applicability conditions and expected behaviors, assigning Pass or Fail verdicts accordingly. Validation experiments show high agreement with human judgments, confirming the method’s reliability.
Applying this approach to SkillsBench, the authors find that only about 39% to 46% of documented skill behaviors are covered in current trajectories, revealing a substantial under-testing issue. Moreover, analyzing failed constraints—those exercised but violated—provides actionable insights. By emphasizing the original instructions associated with these failed SBCs, the authors strengthen the skills, leading to a 16% increase in task success across multiple models.
This work significantly advances the evaluation of agent skills, offering a fine-grained, scalable, and diagnostic tool that bridges the gap between task success and skill-level verification. It opens new avenues for systematic skill development, continuous improvement, and trustworthy deployment of AI systems in complex real-world scenarios. Future efforts will focus on multi-modal data integration and automated constraint refinement to further enhance the framework’s robustness and applicability.
Deep Dive
Abstract
Agent skills encode reusable procedural knowledge for large language model (LLM) agents, and existing benchmarks show that such skills can improve task-level performance. However, a task outcome does not reveal which parts of a reusable skill were exercised, nor whether the agent followed the relevant skill instructions when those parts were exercised. This gap makes it unclear whether a skill has been adequately tested, or whether observed task failures provide actionable evidence for improving agent skill effectiveness. To fill this gap, we introduce skill coverage, a trajectory-based test-adequacy metric for reusable agent skills. Our framework extracts skill behavior constraints from each skill, translating natural-language skill instructions into semi-structured constraints that specify the expected agent behavior under particular conditions. It then determines whether each constraint is covered by an agent trajectory and, for covered constraints, assigns a Pass or Fail verdict according to the agent behavior. We apply this framework to SkillsBench. The results show that agent trajectories on the benchmark leaderboard cover only 38.66 to 45.51% of the extracted skill behavior constraints on average. We then use Fail verdicts to strengthen the corresponding skill content only by emphasizing the original instructions that the agent failed to follow, and run the same tasks with the strengthened skills. This emphasis yields an average 16.0% recovery rate of the failed tasks across the five agent-model rows. These results show that skill coverage is both a test-adequacy metric and a fine-grained signal for observing skill-use behavior. In failed tasks, failed constraint labels provide actionable evidence for improving agent skill effectiveness. A project website accompanies the paper.