Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale
SkillScan framework reveals 26.1% vulnerabilities in AI skill markets, urging enhanced security vetting.
Key Findings
Methodology
The study employs the SkillScan framework, integrating static analysis with LLM-based semantic classification, to systematically analyze 31,132 skills from two major marketplaces. SkillScan identifies 14 distinct vulnerability patterns across four categories: prompt injection, data exfiltration, privilege escalation, and supply chain risks.
Key Results
- The study finds 26.1% of skills contain at least one vulnerability, with data exfiltration at 13.3% and privilege escalation at 11.8%. High-severity patterns account for 5.2%, indicating malicious intent.
- Skills bundling executable scripts are 2.12 times more likely to contain vulnerabilities than instruction-only skills (OR=2.12, p<0.001).
- The study provides a vulnerability taxonomy based on 8,126 vulnerable skills and validates the detection methodology with 86.7% precision and 82.5% recall.
Significance
This study is the first large-scale revelation of security vulnerabilities in AI skill markets, emphasizing the necessity for mandatory security vetting before skill execution. The findings are significant for academia and industry, particularly in enhancing AI systems' security and scalability.
Technical Contribution
The study develops the SkillScan detection framework, combining static analysis and LLM-based semantic classification, offering a novel vulnerability detection method. It also creates an open dataset and detection toolkit to support future research.
Novelty
This is the first large-scale security analysis of AI skill markets, revealing implicit trust issues in skill execution. Unlike previous studies, this research systematically analyzes the code and instructions of skills, not just model behavior.
Limitations
- The study may have survivorship bias as malicious skills might be removed by platforms post-detection, affecting result accuracy.
- The current detection framework may not identify all potential vulnerability patterns, especially complex multi-stage attacks.
Future Work
Future research could focus on longitudinal analysis of skill removal patterns to quantify survivorship bias. Additionally, more complex detection frameworks could be developed to identify multi-stage attacks and complex vulnerability patterns.
AI Executive Summary
With the rise of AI agent frameworks, skills have become crucial modules for extending agent capabilities. However, this architecture introduces significant security risks. Researchers conducted the first large-scale security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces and systematically analyzing 31,132 using the SkillScan framework. The results reveal that 26.1% of skills contain at least one vulnerability, with data exfiltration and privilege escalation being the most common issues. The study also finds that skills bundling executable scripts are more prone to vulnerabilities than instruction-only skills. To address these risks, the study recommends mandatory security vetting before skill execution and the introduction of capability-based permission systems. The technical contributions include the development of the SkillScan detection framework and the creation of an open dataset and detection toolkit to support future research. Although the study reveals security issues in skill markets, further research is needed to address complex multi-stage attacks and potential survivorship bias.
Deep Analysis
Background
AI agent frameworks have rapidly evolved, with skills as modular capability extension packages allowing agents to perform specific tasks without bloating core functionalities. However, this flexibility also introduces security risks, as skills are often executed with implicit trust and minimal security vetting. Major platforms like Claude Code and Codex CLI have adopted this architecture, and a plethora of third-party skills have emerged in the market, increasing the potential attack surface.
Core Problem
The AI skill market lacks systematic security vetting, leading to significant security vulnerabilities during skill execution. Skills can bundle arbitrary executable code, potentially causing data exfiltration, privilege escalation, and other issues. Existing research mainly focuses on model behavior, neglecting the security of skill code and instructions.
Innovation
This study is the first to systematically analyze security vulnerabilities in AI skill markets, developing the SkillScan framework that integrates static analysis with LLM-based semantic classification to identify 14 distinct vulnerability patterns. The study also creates a vulnerability taxonomy based on 8,126 vulnerable skills, providing a foundation for future research.
Methodology
- �� Data Collection: Gathered 42,447 skills from two major marketplaces.
- �� Filtering and Deduplication: Analyzed 31,132 unique skills.
- �� Detection with SkillScan: Integrated static analysis and LLM-based semantic classification.
- �� Manual Annotation: Annotated 500 skills to build a benchmark dataset.
- �� Result Analysis: Identified 14 vulnerability patterns and calculated vulnerability rates.
Experiments
The experimental design includes data collection from two major marketplaces and multi-stage detection using the SkillScan framework. The study analyzes 31,132 skills across eight functional categories. By manually annotating 500 skills, a benchmark dataset is established to validate the detection method's precision and recall.
Results
Results show that 26.1% of skills contain at least one vulnerability, with data exfiltration and privilege escalation being the most common issues. The study also finds that skills bundling executable scripts are more prone to vulnerabilities. The SkillScan framework achieves 86.7% precision and 82.5% recall.
Applications
The findings can be used to develop safer AI skill markets, supporting platforms in conducting mandatory security vetting before skill execution. The study also provides an open dataset and detection toolkit for future security research.
Limitations & Outlook
The study may have survivorship bias as malicious skills might be removed by platforms post-detection. Additionally, the current detection framework may not identify all potential vulnerability patterns, especially complex multi-stage attacks. Future research could focus on longitudinal analysis of skill removal patterns to quantify survivorship bias.
Plain Language Accessible to non-experts
Imagine a large supermarket where AI skills are like products on the shelves. Each product has its own instructions and usage methods, but not every product undergoes rigorous safety checks. Some products might pose unexpected risks, such as data leaks or privilege escalation. Researchers developed a tool called SkillScan, like a supermarket security scanner, capable of identifying these potential risks, ensuring each product is thoroughly vetted before being shelved.
ELI14 Explained like you're 14
Imagine you're playing a game with lots of different skills to use. But some skills might secretly steal your game data or make you lose control of the game. Researchers are like game admins, and they've developed a tool called SkillScan to help detect these dangerous skills, ensuring the game's safety. Now you can use skills without worrying about being hacked!
Glossary
SkillScan
A multi-stage framework for detecting vulnerabilities in AI skills, integrating static analysis and semantic classification.
Used to identify security vulnerabilities in AI skills.
Prompt Injection
Manipulating instructions to bypass safety controls or execute unintended actions.
One of the vulnerability patterns that may be present in skills.
Data Exfiltration
Unauthorized data transmission to external parties.
A common security risk in skills.
Privilege Escalation
Skills gaining access beyond their stated purpose.
A potential vulnerability pattern in skills.
Supply Chain Risks
Risks of introducing malicious functionality through dependencies or remote code.
Security issues that may arise post-skill execution.
Open Questions Unanswered questions from this research
- 1 How to effectively detect complex multi-stage attacks? Current methods may not identify all vulnerabilities.
- 2 How to conduct longitudinal analysis of skill removal patterns to quantify survivorship bias?
Applications
Immediate Applications
Skill Market Security Vetting
Platforms can conduct mandatory security vetting before skill execution to reduce potential risks.
Long-term Vision
AI System Security Enhancement
By developing more complex detection frameworks, identifying multi-stage attacks and complex vulnerability patterns, the overall security of AI systems can be improved.
Abstract
The rise of AI agent frameworks has introduced agent skills, modular packages containing instructions and executable code that dynamically extend agent capabilities. While this architecture enables powerful customization, skills execute with implicit trust and minimal vetting, creating a significant yet uncharacterized attack surface. We conduct the first large-scale empirical security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces and systematically analyzing 31,132 using SkillScan, a multi-stage detection framework integrating static analysis with LLM-based semantic classification. Our findings reveal pervasive security risks: 26.1% of skills contain at least one vulnerability, spanning 14 distinct patterns across four categories: prompt injection, data exfiltration, privilege escalation, and supply chain risks. Data exfiltration (13.3%) and privilege escalation (11.8%) are most prevalent, while 5.2% of skills exhibit high-severity patterns strongly suggesting malicious intent. We find that skills bundling executable scripts are 2.12x more likely to contain vulnerabilities than instruction-only skills (OR=2.12, p<0.001). Our contributions include: (1) a grounded vulnerability taxonomy derived from 8,126 vulnerable skills, (2) a validated detection methodology achieving 86.7% precision and 82.5% recall, and (3) an open dataset and detection toolkit to support future research. These results demonstrate an urgent need for capability-based permission systems and mandatory security vetting before this attack vector is further exploited.