SkillMutator: Benchmarking and Defending Language-and-Code Cross-modal Attacks on LLM Agent Skills
SkillMutator enhances LLM agent skill security detection to 88.2% across 13 attack categories.
Key Findings
Methodology
SkillMutator simulates adversarial mutation processes on LLM agent skills, generating malicious skill packages across 13 attack categories. It employs a four-phase reasoning-trajectory distillation framework to distill frontier-teacher model reasoning into smaller open-weight models, creating a locally deployable scanner.
Key Results
- On the strongest SkillMutator subset, the Qwen2.5-Coder-7B-Instruct model improved detection from 17.1% to 88.2%, surpassing GPT-4o-mini's 23.7% and GPT-5.4-mini's 79.0%.
- Existing open-source and commercial scanners detect only 2%-8% and 9%-17% of the 187 mutated skill scenarios generated by SkillMutator.
- The four-phase reasoning framework successfully distills frontier-level detection capabilities into smaller models, avoiding high API costs.
Significance
This research systematically measures the threat of language-and-code cross-modal attacks and provides an effective benchmark through SkillMutator, significantly enhancing the security detection capabilities of LLM agent skills. It offers a new direction for future AI security research, particularly in achieving efficient local detection without relying on costly frontier models.
Technical Contribution
SkillMutator introduces the first benchmark for detecting language-and-code cross-modal attacks and proposes a four-phase reasoning-trajectory distillation framework, enabling small open-weight models to perform cross-modal semantic detection. This approach achieves efficient local scanning without third-party models.
Novelty
This is the first benchmark for cross-modal attacks on LLM agent skills, providing a systematic measurement framework through adversarial mutation processes, significantly enhancing detection rates.
Limitations
- SkillMutator's detection rates for certain attack categories still have room for improvement, especially in complex cross-modal interaction scenarios.
- The current framework relies on existing attack category definitions, which may not cover future novel attacks.
Future Work
Future research could expand SkillMutator's attack categories, explore more complex cross-modal interaction scenarios, and optimize the existing detection framework to improve adaptability to novel attacks.
AI Executive Summary
As large language model (LLM) agent skills become widely used, their security issues are increasingly prominent. Existing skill scanners perform poorly in detecting language-and-code cross-modal attacks, identifying only a small fraction of attacks. To address this, SkillMutator was developed. It simulates adversarial mutation processes, generating malicious skill packages across 13 attack categories, and employs a four-phase reasoning-trajectory distillation framework to distill frontier-teacher model reasoning into smaller open-weight models, creating a locally deployable scanner. On the strongest SkillMutator subset, the Qwen2.5-Coder-7B-Instruct model improved detection from 17.1% to 88.2%, surpassing GPT-4o-mini's 23.7% and GPT-5.4-mini's 79.0%. These results demonstrate that practical defense against cross-modal attacks is feasible without relying on costly frontier models. The introduction of SkillMutator offers a new direction for AI security research, particularly in achieving efficient local detection without third-party models.
Deep Analysis
Background
As large language model (LLM) agent skills become widely used, their security issues are increasingly prominent. Existing skill scanners perform poorly in detecting language-and-code cross-modal attacks, identifying only a small fraction of attacks. To address this, SkillMutator was developed. It simulates adversarial mutation processes, generating malicious skill packages across 13 attack categories, and employs a four-phase reasoning-trajectory distillation framework to distill frontier-teacher model reasoning into smaller open-weight models, creating a locally deployable scanner.
Core Problem
Existing skill scanners perform poorly in detecting language-and-code cross-modal attacks, identifying only a small fraction of attacks. These attacks exploit the interaction between natural language instructions and executable code, making it difficult for existing unimodal analysis methods to effectively detect them.
Innovation
SkillMutator simulates adversarial mutation processes, generating malicious skill packages across 13 attack categories, and employs a four-phase reasoning-trajectory distillation framework to distill frontier-teacher model reasoning into smaller open-weight models, creating a locally deployable scanner.
Methodology
- �� SkillMutator simulates adversarial mutation processes, generating malicious skill packages across 13 attack categories.
- �� Employs a four-phase reasoning-trajectory distillation framework to distill frontier-teacher model reasoning into smaller open-weight models.
- �� Creates a locally deployable scanner, avoiding high API costs.
Experiments
On the strongest SkillMutator subset, the Qwen2.5-Coder-7B-Instruct model improved detection from 17.1% to 88.2%, surpassing GPT-4o-mini's 23.7% and GPT-5.4-mini's 79.0%. These results demonstrate that practical defense against cross-modal attacks is feasible without relying on costly frontier models.
Results
On the strongest SkillMutator subset, the Qwen2.5-Coder-7B-Instruct model improved detection from 17.1% to 88.2%, surpassing GPT-4o-mini's 23.7% and GPT-5.4-mini's 79.0%. These results demonstrate that practical defense against cross-modal attacks is feasible without relying on costly frontier models.
Applications
The introduction of SkillMutator offers a new direction for AI security research, particularly in achieving efficient local detection without third-party models.
Limitations & Outlook
SkillMutator's detection rates for certain attack categories still have room for improvement, especially in complex cross-modal interaction scenarios. The current framework relies on existing attack category definitions, which may not cover future novel attacks.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and SkillMutator is like a smart assistant that not only helps you identify ingredients but also tells you which ones might be problematic. It analyzes the recipe (natural language instructions) and the ingredients (executable code) combination to ensure your dish is safe and delicious. Even if some ingredients seem fine, if combined with certain instructions, they might pose a risk, and SkillMutator will warn you in advance.
ELI14 Explained like you're 14
Imagine you're playing a game, and your task is to protect a castle from invaders. SkillMutator is like a super detective that can spot characters that seem harmless but might actually be dangerous. Even if some characters look friendly, if combined with certain items, they might pose a risk, and SkillMutator will warn you in advance.
Glossary
SkillMutator
A benchmark tool for detecting cross-modal attacks on LLM agent skills.
Used to generate and detect malicious skill packages.
Cross-modal Attack
An attack that exploits the interaction between natural language instructions and executable code.
Common in LLM agent skills.
Reasoning Trajectory Distillation
A technique to distill complex model reasoning into smaller models.
Used to enhance local scanner detection capabilities.
Qwen2.5-Coder-7B-Instruct
A small open-weight model for cross-modal semantic detection.
Used in SkillMutator to improve detection rates.
Frontier Teacher Model
A complex model used to guide reasoning trajectory distillation.
Used in SkillMutator to generate training data.
Open Questions Unanswered questions from this research
- 1 How to further improve cross-modal attack detection rates without increasing computational costs?
- 2 How to expand SkillMutator to cover future novel attacks?
Applications
Immediate Applications
Enterprise Security Detection
Enterprises can use SkillMutator to detect and defend against potential cross-modal attacks, protecting internal systems.
Long-term Vision
AI Security Standard Development
SkillMutator can provide a reference for developing AI security standards, promoting industry security development.
Abstract
Large language model (LLM) agents increasingly extend their capabilities at runtime by loading Agent Skills, which pair natural-language specifications (SKILL.md) with executable scripts and resources. Because a skill's behavior relies on both natural-language instructions and executable code, assessing its safety requires cross-modal reasoning, creating a new language-and-code attack surface. Attackers can present a benign workflow in SKILL.md while embedding implicit directives that steer the agent to exfiltrate sensitive files, even if the scripts appear harmless. This attack surface remains understudied; prior work treats skills merely as prompt-injection vectors or static code artifacts, leaving attacks emerging from cross-modal interactions largely unmeasured. In our evaluation, open-source and commercial skill scanners detect only 2%-8% and 9%-17% of such attacks, respectively. To address this gap, we introduce SkillMutator, the first benchmark for install-time detection of language-and-code cross-modal attacks on Agent Skills. It emulates an adversarial mutation process across 13 attack categories, iteratively refining malicious skills using scanner feedback to make injected behaviors indistinguishable from legitimate workflows. We further propose a four-phase reasoning-trajectory distillation framework to distill frontier-teacher traces into smaller open-weight models. This produces a locally deployable scanner avoiding third-party data exposure and excessive API costs. On the strongest SkillMutator subset (n=76), our distilled model (Qwen2.5-Coder-7B-Instruct) improves detection from 17.1% to 88.2%, surpassing GPT-4o-mini (23.7%) and GPT-5.4-mini (79.0%), and reaching frontier-level GPT-5.4 (86.8%). These results show practical defense against cross-modal attacks is feasible without relying on costly frontier models.