SkillEval: Decomposing Agent Skill Quality into Interpretable Signals
SkillEval learns interpretable linear directions in model embeddings to evaluate four skill document dimensions, correlating with downstream task performance.
Key Findings
Methodology
SkillEval constructs each quality dimension as a fixed linear direction in the hidden state space of a frozen language model (Qwen3.5-9B). Using contrastive positive-negative skill pairs, it learns these directions via difference pooling and orthogonalization against confounding biases like document length. During evaluation, each skill embedding is projected onto these directions, standardized, and bias-corrected, producing interpretable scores. The framework was validated on 518 skills, demonstrating high separation between positive and negative samples across six dimensions, and strong correlation (Pearson r=0.779) with downstream task uplift. This approach enables efficient, transparent, multi-dimensional skill quality assessment without model fine-tuning.
Key Results
- On a validation set of 518 skills, positive skills scored significantly higher than negatives across all six metrics (e.g., A2: 0.98 vs. 0.96), confirming the learned directions' validity. The scores correlated strongly with pass-rate uplift in downstream tasks, with Pearson r=0.779, indicating high practical relevance. Bias analysis showed minimal influence from document length, thanks to orthogonalization, ensuring semantic focus. Skill revisions guided by these scores improved downstream success rates by an average of 29.5 percentage points, demonstrating the framework’s utility for skill enhancement.
- The metrics maintained high consistency across different backbone models (Qwen and Llama series), with correlation coefficients exceeding 0.91, confirming robustness. Task-specific utility varied significantly, emphasizing the importance of multi-dimensional, generalizable quality signals. The framework also facilitated targeted skill revisions, leading to substantial performance gains, and provided diagnostic insights into weaknesses, enabling precise improvements.
Significance
This work advances skill evaluation from task-dependent, black-box metrics to interpretable, intrinsic quality signals rooted in model internal representations. It addresses key challenges in scalable, reliable skill management by providing a transparent, multi-faceted assessment tool that correlates with real-world utility. Such a framework supports automated skill curation, diagnosis, and refinement, crucial for deploying large-scale autonomous agents and knowledge bases. Its interpretability fosters trust and facilitates targeted improvements, bridging the gap between internal model understanding and practical skill development, thus contributing significantly to AI system robustness and transparency.
Technical Contribution
SkillEval introduces a novel contrastive learning approach to extract fixed, interpretable linear directions for each skill quality dimension directly from model hidden states, without requiring model fine-tuning. The orthogonalization against bias directions ensures semantic purity. Its multi-dimensional scoring enables detailed diagnostics and targeted revisions. Compared to prior black-box or single-metric methods, this approach offers explainability, efficiency, and robustness, making it suitable for large-scale, real-time skill assessment. The framework's compatibility with multiple backbone models and its strong correlation with downstream utility mark significant technical progress in intrinsic evaluation.
Novelty
This is the first systematic application of linear directions in model internal representations for multi-dimensional skill quality evaluation. Unlike traditional task-based or black-box metrics, SkillEval leverages contrastive learning to derive explicit, interpretable semantic axes, enabling fine-grained diagnostics. Its bias correction and multi-model robustness further distinguish it from prior work. This approach opens new avenues for explainable AI evaluation, integrating internal model understanding with practical performance metrics.
Limitations
- The reliance on predefined positive-negative pairs may introduce sampling bias, limiting generalization to unseen or diverse skills. The linear assumption might oversimplify complex semantic relationships, reducing sensitivity in nuanced cases. In noisy or poorly formatted documents, the scores' stability and reliability could diminish, necessitating more sophisticated, possibly non-linear, modeling approaches in future work.
Future Work
Future directions include integrating non-linear or multi-modal representations to capture richer semantic nuances, expanding the positive-negative pair construction via active learning, and applying the framework to dynamic skill repositories for continuous evaluation. Exploring domain adaptation techniques will enhance robustness across diverse applications. Additionally, combining this approach with human-in-the-loop feedback could further refine interpretability and accuracy, supporting scalable, automated skill ecosystems for increasingly autonomous AI systems.
AI Executive Summary
SkillEval introduces a novel interpretability-driven framework for assessing agent skill documents by decomposing their quality into four distinct, measurable dimensions. Traditional evaluation methods rely heavily on downstream task performance, which can be environment-dependent and insufficient for comprehensive quality assessment. In contrast, SkillEval leverages the internal representations of a frozen language model, specifically the hidden states of Qwen3.5-9B, to learn fixed linear directions corresponding to applicability, content quality, execution guidance, and robustness. These directions are derived through contrastive learning on controlled positive-negative skill pairs, ensuring that each dimension captures a specific semantic property. During evaluation, each skill embedding is projected onto these directions, standardized, and bias-corrected to produce interpretable scores that reflect intrinsic qualities. Extensive experiments on 518 skills demonstrated that these scores reliably distinguish high- and low-quality skills, with clear separation in the validation set (e.g., A2: 0.98 vs. 0.96). Moreover, the scores correlate strongly with downstream task success uplift (Pearson r=0.779), confirming their predictive utility. The bias correction mechanism effectively reduces influence from document length and formatting, ensuring semantic purity. Applying these scores for skill diagnosis and targeted revision led to significant improvements in skill quality and downstream performance, with an average pass rate increase of 29.5 percentage points. The framework's robustness was validated across multiple backbone models, maintaining high consistency. Overall, SkillEval offers a scalable, transparent, and effective tool for intrinsic skill evaluation, diagnosis, and refinement, paving the way for more reliable and interpretable autonomous systems. Future work will explore non-linear extensions, multi-modal integration, and continuous skill management to further enhance its applicability and robustness.
Deep Analysis
Background
随着人工智能和自主系统的快速发展，技能文档作为知识存储和复用的核心载体，逐渐成为研究焦点。早期工作如Liang等（2026a）提出技能包的构建与管理，随后多项研究关注任务驱动的技能评估（Li et al., 2026b; Han et al., 2026），但其局限在于依赖实际任务执行，成本高且受环境影响。近年来，intrinsic评估方法逐渐兴起，试图从文档内容和模型内部表征出发，评估技能的内在质量（Zhong et al., 2026; Hong et al., 2026）。这些方法旨在实现高效、可解释的技能质量量化，减少对任务环境的依赖，为大规模技能库的自动管理提供可能。
Core Problem
现有技能评价多依赖于下游任务表现，难以全面反映技能的内在属性，且受任务场景限制，难以实现跨任务的普适性。此外，缺乏可解释的指标，难以指导技能改进。文档长度、格式等非语义特征对评估结果的干扰，也影响指标的可靠性。如何在模型内部空间中学习具有语义解释的多维指标，成为亟待解决的问题。这限制了技能评估的效率和可操作性，阻碍了技能生态系统的健康发展。
Innovation
本研究提出SkillEval框架，创新点包括：1）利用模型内部隐藏状态空间，通过对比学习构建每个质量属性的线性方向，提供可解释的多维指标；2）引入偏差校正机制，减少文档非语义特征对评分的影响，确保指标反映目标属性；3）实现无需微调模型，仅通过固定方向投影快速评估技能质量。该方法结合多维度分析，细粒度刻画技能的适用性、内容质量、执行指导和鲁棒性，为技能评估提供了全新思路。
Methodology
- �� 采集正负技能对，构建目标属性差异明显的样本。• 使用预训练模型（Qwen3.5-9B）提取隐藏状态，选择不同池化策略（描述、正文、最后一词）获得技能表示。• 计算表示差异，学习每个属性的线性方向，结合偏差校正，去除非语义偏差。• 固定方向后，将新技能表示投影到方向上，得到属性得分。• 标准化得分，确保不同技能间的可比性。• 通过偏差校正，减少文档长度和格式的影响，提升指标的语义纯粹性。• 在518技能样本上验证指标的区分能力和与下游任务的相关性，确保指标的实用性。
Experiments
采用公开技能数据集，构建正负对进行训练和验证。利用Qwen3.5-9B模型提取隐藏状态，学习六个维度的线性方向。验证指标在未见技能上的区分能力和与任务成功率的相关性。偏差校正确保评分不受文档长度影响。模型鲁棒性测试验证不同模型架构下指标一致性。最后，将指标应用于技能修订，验证其在提升技能质量和任务成功率中的指导作用。
Results
指标在518技能验证集中，正技能平均得分高于负技能（如A2: 0.98 vs. 0.96），差异显著。不同维度得分与下游任务提升（Pearson r=0.779）高度相关，验证指标实用性。偏差校正后，评分与文档长度关系微弱，确保语义纯粹。技能修订后，任务成功率平均提升29.5个百分点，验证指标指导效果。模型鲁棒性测试显示，不同模型架构下指标相关性超过0.91，表明方法具有良好的泛化能力。
Applications
该方法适用于大规模技能库的自动评估与优化，支持技能的快速诊断和针对性修订。可广泛应用于智能助理、自动化流程、企业知识管理等场景，提升技能的质量控制效率。未来结合主动学习和多模态信息，推动技能生态系统的智能化管理，实现持续优化和自动修正。
Limitations & Outlook
当前方法依赖正负技能对的构建，可能受样本偏差影响，影响泛化能力。线性方向假设在某些复杂语义关系中可能不足，限制了评估表达能力。对于极端或噪声较多的技能文档，评分稳定性仍需验证，未来需引入非线性模型或多模态融合技术以增强鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房里做菜，每个菜谱就像一份技能文档。传统评价只看菜做得好不好（味道是否佳），但这不能告诉你菜谱哪里可以改进。SkillEval就像一个聪明的厨师助手，它会分析菜谱的不同方面，比如用料是否合理、步骤是否清晰、是否适合不同厨房设备、以及菜的抗变异能力。它通过观察很多菜谱的例子，学会了每个方面的“好”与“不好”的线性指标。这样，无论你给它什么菜谱，它都能告诉你在哪些方面可以改进，帮助你写出更好的菜谱。这个方法让评价变得透明、可解释，也能指导你不断优化菜谱，做出更受欢迎的菜肴。
ELI14 Explained like you're 14
想象你在学校写作文，老师要你写得更棒。以前老师只看最后得分，觉得作文好就高分，不好就低分。这虽然简单，但不能告诉你具体哪里写得不好。SkillEval就像一个聪明的朋友，它会帮你分析作文的不同部分，比如开头是否吸引人、内容是否丰富、结尾是否有力、语法是否正确。它通过学习很多优秀和不好的作文，找到每个部分的“好”与“不好”的线性指标。每次你写完作文，它就能告诉你在哪些方面可以改进，比如“开头可以更吸引人”或“结尾需要更有力”。这样，你就可以有针对性地修改，写出更棒的作文。这个工具让评价变得更具体、更公平，也能帮你不断变得更厉害！
Abstract
Agent skills provide reusable procedural knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality by testing whether a skill improves performance on specific downstream tasks. However, a reusable skill may apply to multiple task scenarios. Downstream evaluation mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspect of the skill should be improved. We find that general properties of the \texttt{SKILL.md} document play an important role in skill quality. To evaluate these properties, we propose \textbf{SkillEval}, an interpretable framework for document-level skill evaluation. SkillEval evaluates each property using a fixed and inspectable scoring direction, producing interpretable scores. It further measures and reduces the influence of unrelated document features, such as length and formatting, so that each score captures its intended semantic property more specifically. Specifically, SkillEval learns an interpretable direction for each quality property from controlled positive--negative skill pairs in the hidden representation space of the model, and scores a new skill by projecting its representation onto these fixed directions. We use SkillEval to evaluate skills in controlled quality tests and show that SkillEval reliably distinguishes skills of different quality. In addition, SkillEval scores closely reflect downstream task performance, providing an early indication of whether a skill is likely to help an agent complete a task. We further explore SkillEval for diagnosing weaknesses in skill documents and guiding targeted revisions. The revised skills improve the targeted properties and achieve higher pass rates on downstream tasks.