Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise
Inference-time backdoor via chat template reduces accuracy from 90% to 15%.
Key Findings
Methodology
The paper introduces an inference-time backdoor attack using chat templates. Attackers modify Jinja2-based templates to implant backdoors without altering model weights. This method exploits the template's privileged position between user input and model processing, requiring no access to training pipelines or deployment infrastructure.
Key Results
- At the LLM level, triggered backdoors reduce accuracy from 90% to 15%, with attacker-controlled URL emission success rates over 80%.
- At the agent level, template backdoors hijack tool-use, bypassing all injection defenses.
- In multi-agent systems, a single poisoned artifact compromises real-world deployments.
Significance
This research highlights chat templates as a reliable and currently undefended attack surface in the open-weight AI supply chain. It demonstrates that attackers can implant backdoors without altering model weights or controlling runtime infrastructure, impacting model behavior and security.
Technical Contribution
The technical contribution lies in demonstrating a new method for backdoor attacks without modifying model weights and revealing vulnerabilities undetectable by current security scans. This provides a new perspective on AI model security research.
Novelty
This study is the first to demonstrate the feasibility of inference-time backdoor attacks via chat templates, fundamentally differing from existing methods that rely on training or deployment access.
Limitations
- The method relies on users downloading and using modified model artifacts.
- Existing defenses cannot detect this type of attack.
Future Work
Future research could focus on developing methods to detect and defend against chat template attacks and assess their impact on larger-scale systems.
AI Executive Summary
As open-weight language models are increasingly used in production, security challenges are becoming more prominent. This paper proposes a novel inference-time backdoor attack method by modifying chat templates instead of model weights. Attackers can implant backdoors without altering model weights or controlling runtime infrastructure.
In experiments, researchers evaluated this attack across three deployment tiers. At the LLM level, triggered backdoors reduce accuracy from 90% to 15%, with attacker-controlled URL emission success rates over 80%. At the agent level, template backdoors hijack tool-use, bypassing all injection defenses. In multi-agent systems, a single poisoned artifact can compromise real-world deployments.
These results establish chat templates as a reliable and currently undefended attack surface in the open-weight AI supply chain. Future research could focus on developing methods to detect and defend against chat template attacks and assess their impact on larger-scale systems.
Deep Analysis
Background
The widespread use of open-weight language models brings new security challenges. Traditional backdoor attacks often rely on access to training pipelines or deployment infrastructure, while existing research mainly focuses on the security of model weights.
Core Problem
The core problem is how to achieve backdoor attacks without altering model weights. Existing defenses mainly target attacks during training and deployment phases, overlooking potential risks during inference.
Innovation
The innovation lies in using chat templates as an attack surface by modifying Jinja2-based templates to implant backdoors without altering model weights. This method requires no access to training pipelines or deployment infrastructure.
Methodology
- �� Modify Chat Templates: Attackers extract and modify Jinja2 templates in GGUF format.
- �� Implant Backdoor: Add conditional logic to the template to inject malicious instructions when trigger phrases are detected.
- �� Redistribute Model: Distribute the modified model through public channels.
Experiments
The experimental design includes testing the attack across three deployment tiers. At the LLM level, 18 models were tested; at the agent level, two benchmarks were used; in multi-agent systems, real-world deployments were simulated.
Results
Experimental results show that triggered backdoors significantly reduce model accuracy, with attacker-controlled URL emission success rates over 80%. At the agent level, backdoors can hijack tool-use, bypassing all injection defenses.
Applications
This attack method can be used to assess AI model security, especially in the open-weight model supply chain. It reveals the inadequacies of existing security mechanisms, promoting safer model deployment practices.
Limitations & Outlook
The method relies on users downloading and using modified model artifacts, and existing defenses cannot detect this type of attack. Future research needs to develop new detection and defense methods.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Normally, you'd follow a recipe to prepare each dish. Now, imagine someone secretly added extra steps to your recipe, like adding a special spice at a specific time, but only under certain conditions. This extra step is like a backdoor, only triggered under specific circumstances. The method in this paper is about adding such steps to an AI model's 'recipe' without changing its 'ingredients' (weights).
ELI14 Explained like you're 14
Imagine you're playing a game, and there's a secret level that only appears when you enter a specific password. This secret level is like a backdoor, only triggered under certain conditions. The method in this paper is about adding such secret levels to an AI model without changing its basic settings. Isn't that cool? But it also means we need to be more careful to ensure no one secretly adds unwanted secret levels to our games!
Glossary
Backdoor Attack
An attack method where hidden behaviors are embedded in a model, activated only under specific conditions.
In this paper, backdoor attacks are implemented via chat template modifications.
Chat Template
An executable program that transforms user input before model processing, often implemented in Jinja2.
In this paper, attackers modify chat templates to implant backdoors.
Jinja2
A general-purpose templating language supporting conditionals, loops, and string manipulation.
In this paper, Jinja2 is used to implement chat templates.
GGUF
A single-file format for quantized models, containing model weights and template metadata.
In this paper, attackers modify chat templates in GGUF format.
Inference-Time Attack
An attack conducted during the model inference phase, typically without modifying model weights.
In this paper, inference-time attacks are implemented via chat template modifications.
Open Questions Unanswered questions from this research
- 1 How to detect and defend against chat template attacks without affecting model performance?
- 2 Why can't existing security scans detect this type of attack?
- 3 How to assess the impact of this attack on larger-scale systems?
Applications
Immediate Applications
Model Security Assessment
Can be used to assess AI model security, especially in the open-weight model supply chain.
Long-term Vision
Secure Model Deployment
Promotes safer model deployment practices, developing new detection and defense methods.
Abstract
Open-weight language models are increasingly used in production settings, raising new security challenges. One prominent threat is backdoor attacks, in which adversaries embed hidden behaviors that activate under specific conditions. Previous work has assumed that adversaries have access to training pipelines or deployment infrastructure. We propose a novel attack surface requiring neither: the "chat template". Chat templates are executable programs invoked at every inference call, often implemented in Jinja2, that occupy a privileged position between user input and model processing. We show that an adversary who distributes a model with a maliciously modified template can implant an inference-time backdoor without modifying model weights, poisoning training data, or controlling runtime infrastructure. We evaluate this attack across three deployment tiers. At the LLM level, triggered backdoors reduce factual accuracy from 90% to 15% on average and induce attacker-controlled URL emission with success rates exceeding 80%, while benign inputs show no measurable degradation; these results hold across eighteen models. At the agent level, template backdoors hijack tool-use across two benchmarks spanning 3,868 episodes, bypassing every tested injection defense offered by the benchmarks while remaining fully dormant absent the trigger. At the multi-agent system level, we demonstrate how a single poisoned artifact compromises a real-world agentic deployment and propagates supply-chain code poisoning downstream. The poisoned artifacts evade all security scans on the largest open model distribution platform; and because the payload is rendered by the template before user input is processed, it is architecturally unreachable by input-level defenses such as prompt injection guardrails. These results establish chat templates as a reliable and undefended attack in the open-weight AI supply chain.