SkillShield: Prompt-Space Security Skills for LLM Coding Agents
SkillShield synthesizes offline security skills embedded in system prompts, reducing malicious code generation success rate from 43.6% to 14.5% across six LLMs.
Key Findings
Methodology
The approach involves offline extraction of security policies from known attack data and agent failure logs, then encoding these as natural-language security skills within the system prompt. These skills are fixed, threat-specific, and cover all classes, subsets, or individual threats, without requiring runtime classification or routing. Experiments on six large language models (e.g., GPT-4, Llama 2, Claude) demonstrate that embedding these skills reduces malware severity from 3.37 to 0.58 and attack success rate from 43.6% to 14.5%. The method balances prompt space constraints with threat coverage, optimizing security effectiveness.
Key Results
- Default all-classes skill reduces malware severity from 3.37 to 0.58, attack success rate from 43.6% to 14.5%.
- Per-bundle and per-class configurations further improve defenses, lowering success rates to 36.2% and 14.5%.
- Under adaptive jailbreaks, SkillShield maintains superiority over baselines, with a mean safety refusal rate of 0.14%.
Significance
This work introduces a novel, cost-effective method for embedding threat-specific security policies directly into the prompt, bypassing the need for weight modification or runtime checks. It addresses critical gaps in current defenses by enabling static, threat-aware control over model actions, suitable for API-only deployment scenarios. The approach enhances robustness against malicious prompts and malware generation, providing a scalable solution for real-world AI safety challenges.
Technical Contribution
The core innovation lies in the offline synthesis of security skills from attack data, formatted as natural-language policies embedded in the system prompt. This method leverages attack signatures and failure analyses to generate threat-specific detection and refusal rules, which are then fixed in the prompt. The multi-scope configuration (all-class, subset, single-class) allows flexible deployment, balancing prompt space and threat coverage. The approach avoids retraining or additional runtime components, making it practical for API deployment.
Novelty
This is the first systematic study of threat-specific, fixed-budget security policies synthesized offline and embedded in prompt space for LLM agents. Unlike prior boundary reminders or runtime checks, it emphasizes proactive, static policy injection based on attack data, offering a new paradigm for prompt-based security. The multi-scope design and threat-specific synthesis distinguish it from general-purpose safety prompts.
Limitations
- The approach depends heavily on the quality and comprehensiveness of attack data; unseen threats may bypass the fixed policies.
- Prompt space constraints limit the granularity of threat coverage, especially in broader scopes, potentially reducing effectiveness against complex attacks.
- It assumes static threat landscapes; adaptive or evolving attacks may require frequent skill updates, which are not addressed in this study.
Future Work
Future research will explore dynamic updating of security skills, integrating real-time threat intelligence, and extending to multi-modal inputs. Combining reinforcement learning with offline synthesis could optimize policy effectiveness. Additionally, expanding threat coverage and improving prompt space efficiency will be key to handling evolving attack vectors.
AI Executive Summary
The rapid adoption of large language models (LLMs) in code generation and system automation has exposed significant security vulnerabilities. Malicious prompts can lead models to execute harmful commands or produce malicious code, threatening data integrity and system safety. Existing defenses, such as model fine-tuning, input filtering, and runtime monitoring, each have limitations: fine-tuning is costly and inflexible, filters can be bypassed, and runtime checks introduce latency and complexity.
In response, this paper introduces SkillShield, a novel approach that embeds threat-specific security policies directly into the system prompt. By synthesizing these policies offline from attack data and failure logs, SkillShield creates fixed, threat-aware safety skills that remain active throughout the interaction. This method circumvents the need for model retraining or runtime classification, making it highly suitable for API deployment environments.
Experimental results across six large language models demonstrate the effectiveness of this approach. The default all-classes security skill reduces malware severity from 3.37 to 0.58 and attack success rate from 43.6% to 14.5%. Further, deploying class-specific or subset-specific skills enhances protection, maintaining robustness even under adaptive jailbreak attacks. Importantly, the approach introduces minimal safety-related false refusals, with an average rate of just 0.14% across benign tasks.
This work signifies a major step forward in prompt-based security strategies. It offers a scalable, low-cost, and threat-specific defense mechanism that can be easily integrated into existing systems. By focusing on offline synthesis and fixed prompt embedding, SkillShield provides a practical solution to the evolving challenges of AI safety, with potential applications in enterprise AI services, open-source tools, and cloud APIs.
Looking ahead, future directions include dynamic updating of security skills, integrating multi-modal threat intelligence, and expanding threat coverage to address emerging attack vectors. This approach opens new avenues for proactive, threat-aware AI safety, ensuring more secure deployment of powerful language models in real-world scenarios.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂工作，工厂每天都在生产各种商品。有时候，有些坏人会偷偷带来坏货，试图让工厂生产出有害的商品，甚至可能造成工厂的损失。工厂的管理者希望提前制定一些安全规则，告诉工人遇到坏货时要拒绝生产，不让坏货进入生产线。SkillShield就像这些提前写好的安全规则，它们被放在工厂的操作手册里，工人在工作时会遵守这些规则，自动识别并拒绝那些危险的请求。这样，即使坏人试图欺骗工人，也难以成功，工厂的安全就得到保障。这些规则不需要每次都重新调整，只要提前写好，工人就能一直遵守，保证工厂正常运转。
ELI14 Explained like you're 14
想象你在学校里，有一位老师会提前告诉你哪些行为是不允许的，比如不能偷偷作弊或者做坏事。每次你遇到这些不好的事情时，你都会立刻拒绝，不去做。SkillShield就像老师提前写好的规则卡片，放在你的书包里，每次上学都带着。这样，无论有人试图让你做坏事，你都能马上知道不能做，自己也不会被欺骗。这种方法不用每次都重新教你规则，只要提前准备好，就能一直帮你保护自己，避免陷入麻烦。它就像一份安全指南，让你在学校里更安心、聪明地应对各种情况。
Abstract
A coding agent edits files and executes shell commands with its developer's privileges, allowing malicious requests to translate directly into harmful actions or functional malware. Existing defenses have complementary limitations: weight-level alignment is unavailable to API-only deployers, whereas input filters and execution-boundary monitors require auxiliary classification or checking components along the agent's trajectory. We therefore introduce SkillShield, a system-prompt defense that synthesizes security skills offline from known attacks or recorded agent failures. These skills are injected into the system prompt at session start and remain active throughout the tool-use loop. Unlike a reference monitor, they protect the system by defining the security policies the model should follow during execution. Due to the limited system-prompt space, we examine three fixed-budget provisioning scopes: all-classes, with one skill covering all threat classes, per-bundle, with one skill targeting a related subset, and per-class, with one skill dedicated to a single known class and used as the upper-bound reference. None requires runtime request classification or routing. Across six large language models on RedCode, the default all-classes skill reduces malware-generation severity from 3.37 to 0.58 and achieves a 43.6% execution attack success rate, comparable to Llama Guard 3's 42.7% without its separate 8B classifier. The per-bundle and class-fixed per-class settings further reduce this rate to 36.2% and 14.5%, respectively. Under two non-adaptive jailbreak families, SkillShield continues to outperform all baselines on malware generation. Across 731 benign task descriptions, SkillShield yields a mean safety-refusal rate of 0.14%. These results demonstrate the potential of prompt-space security skills to prevent harmful actions and malware generation for LLM coding agents.