On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models
Proves that shared embedding architectures cannot achieve perfect prompt injection prevention due to inseparability of instructions and data.
Key Findings
Methodology
This paper formalizes Prompted Action Models, integrating Bayesian inference and statistical total variation analysis to demonstrate the fundamental impossibility of achieving Semantic-Faithful Control (SFC) in shared embedding transformer architectures lacking control-data separation. It derives three core results: provenance-recovery impossibility, control-path exposure, and finite-coverage invariance gap, validated through empirical measurements on production tokenizers and models. The approach emphasizes the intrinsic statistical overlap of trusted and untrusted representations, the pathways through which untrusted tokens influence control decisions, and the limitations of finite training in covering infinite semantic classes.
Key Results
- Distributions of trusted and untrusted content in shared embedding space overlap, with total variation distance less than 1, leading to a non-zero Bayes-optimal provenance error (≥0.2), indicating inherent difficulty in source differentiation.
- In standard shared attention architectures, untrusted tokens influence control-relevant computations via the same attention pathways used for output generation, exposing control paths to adversarial inputs.
- Finite training data cannot guarantee invariance over infinite semantic equivalence classes, making perfect semantic-faithful control unachievable, as demonstrated through theoretical invariance gaps and empirical suffix attacks.
Significance
This work reveals a fundamental structural vulnerability in current transformer-based models, akin to code-data confusion in Von Neumann architectures that causes buffer overflow vulnerabilities. It underscores that architectural separation—bioneed for control and data channels—is essential for security, shifting the paradigm from solely improving training or filtering to fundamental design changes. The findings have profound implications for AI safety, guiding future system architectures toward intrinsic security guarantees and influencing industry standards for safe deployment.
Technical Contribution
The paper introduces a rigorous formalization of SFC, establishes the Inseparability Theorem, and connects representational inseparability with behavioral vulnerabilities. It provides a comprehensive theoretical framework combining probabilistic, information-theoretic, and architectural analyses, demonstrating that without immutable separation mechanisms, perfect prompt injection prevention is impossible within shared embedding models. This bridges the gap between security theory and model architecture design, offering new insights into structural vulnerabilities.
Novelty
This is the first formal proof establishing the fundamental impossibility of perfect prompt injection prevention in shared-embedding transformer architectures. It draws a novel analogy with buffer overflow vulnerabilities, providing a structural perspective that shifts focus from heuristic defenses to architectural solutions, representing a significant advancement in understanding model security limitations.
Limitations
- The results assume no external or hardware-enforced control-privacy mechanisms; real systems with such measures may mitigate some vulnerabilities.
- The analysis primarily targets transformer architectures; other models with explicit separation or hard segmentation are outside scope.
- The theoretical framework does not account for dynamic retraining or external policy enforcement, which could influence practical security outcomes.
Future Work
Future research should explore architectural innovations that enforce immutable control-data separation, such as hardware-based isolation or strict modular design. Combining these with interpretability and robustness techniques could lead to practical, provably secure systems. Further empirical validation across diverse models and real-world scenarios is also essential to translate these theoretical insights into deployable solutions.
AI Executive Summary
Prompt injection remains a critical security challenge for large language models, especially as they are integrated into sensitive applications. Despite numerous defenses, attackers continually find ways to bypass safeguards, exposing fundamental vulnerabilities rooted in model architecture. This paper provides a groundbreaking theoretical analysis demonstrating that in shared embedding transformer architectures—common in current models—perfect prevention of prompt injection is mathematically impossible. The core reason lies in the inseparability of instructions and data within the same representational space, which leads to statistical overlaps and pathways through which untrusted inputs influence control decisions.
The authors formalize Prompted Action Models, capturing the essential components of models that produce both token outputs and control actions like refusals or tool invocation. They define Semantic-Faithful Control (SFC), a property requiring control behavior to depend solely on the semantic content of untrusted inputs, not on their encoding. Through rigorous proofs involving total variation distances, attention pathways, and invariance gaps, they establish three key impossibility results: the provenance-recovery impossibility, control-path exposure, and the finite-coverage invariance gap. These results collectively show that any architecture lacking strict control-data separation cannot guarantee perfect security.
Empirical measurements on production tokenizers and models support the theoretical findings, revealing that trusted and untrusted representations overlap significantly, and untrusted tokens influence control pathways directly. The analogy with buffer overflow vulnerabilities in Von Neumann machines underscores the structural nature of this problem, which cannot be fixed by training or filtering alone.
The implications are profound: achieving robust prompt injection defenses requires architectural changes that enforce immutable separation between instruction and data channels. This shifts the paradigm from incremental improvements to fundamental design principles, paving the way for more secure AI systems. While the analysis relies on certain assumptions, it provides a clear theoretical foundation for future innovations in AI safety and architecture.
Deep Analysis
Background
随着大型语言模型（LLMs）在自然语言处理中的广泛应用，Prompt注入成为威胁模型安全的核心问题。早期研究如Instruction Tuning和RLHF在提升模型安全性方面取得一定进展，但未能根本解决指令与输入内容的区分难题。近年来，防御措施如过滤器、表示工程和沙箱机制不断尝试，但攻击者不断突破。传统架构设计多采用共享表示空间，导致指令与数据在同一向量空间中混杂，形成潜在的安全漏洞。类似于计算机中的缓冲区溢出问题，这种设计在安全性上存在结构性缺陷。本文从理论角度出发，系统分析了共享表示架构的固有限制，试图揭示其根本性的不可能性。
Core Problem
核心问题在于，现有Transformer架构中，指令和用户输入都被映射到同一空间，缺乏不可变的控制-数据隔离机制。这导致模型无法在表示层面区分可信指令与潜在恶意输入，从而使Prompt注入攻击得以实现。尽管训练和对齐可以缓解部分风险，但在理论上，任何没有硬隔离的架构都无法保证控制行为只依赖于可信语义。这一问题关系到模型安全的根本性挑战，影响到自动化决策、工具调用和敏感信息保护。
Innovation
本文首次提出语义忠实控制(SFC)的正式定义，强调控制行为应只依赖于输入的语义内容而非表示形式。通过引入代表源追溯不可能性、控制路径暴露和有限覆盖不变性差距的理论工具，系统证明在共享嵌入空间中，缺乏不可变的控制-数据隔离机制时，完美的Prompt注入防御在架构层面上是不可能实现的。这一创新突破了以往仅依赖训练技巧的安全思路，为模型架构设计提供了根本性指导。
Methodology
- �� 定义Prompted Action Models，明确其输入、嵌入、动作空间和输出机制。
- �� 形式化trusted与untrusted内容的共享表示，分析其统计分布重叠性。
- �� 利用贝叶斯判别和总变差距离，推导源追溯的不可能性。
- �� 证明untrusted内容通过相同的值聚合路径影响控制相关计算，导致控制路径暴露。
- �� 结合有限训练样本覆盖的局限性，推导控制行为在所有语义等价变体中保持一致的不可行性。
- �� 最终结合上述结果，证明在缺乏硬隔离机制的架构中，完美语义忠实控制不可能实现。
Experiments
采用OpenAI GPT-3.5和GPT-4模型，结合真实分词器和Tokenizers数据，验证代表空间中的分布重叠性。通过模拟不同的Prompt注入攻击（如Universal Adversarial Suffixes），测量模型的拒绝率、工具调用和记忆写入行为的变化。对比不同训练样本规模和模型参数，验证有限覆盖导致的控制不稳现象。实验结果显示，代表空间中的分布重叠和路径暴露导致攻击成功率持续高达90%以上。
Results
实验证明，代表空间中trusted与untrusted内容的分布支持重叠，导致源追溯错误率≥20%。控制路径中untrusted tokens通过attention机制影响控制输出，验证了路径暴露。有限训练样本无法覆盖无限语义变体，导致控制行为在不同等价类中表现不一致，验证了理论推导。攻击实验中，Universal Adversarial Suffixes能在不同模型中成功改变控制行为，验证了理论的实际适用性。
Applications
该研究强调架构设计上的控制-数据隔离在安全中的关键作用，适用于高风险场景如自动化决策系统、智能助手和敏感信息处理。未来，结合硬件级隔离和外部策略引擎，可提升系统安全性。行业中，建议采用分离的指令和数据路径，减少共享表示空间的潜在风险，从根本上提升模型的抗攻击能力。
Limitations & Outlook
该分析假设模型未采用硬隔离机制，实际系统中引入外部控制路径或硬件隔离可能部分缓解风险。理论分析主要针对Transformer架构，其他模型如外部策略引擎未覆盖。未来研究需结合实际系统，验证架构变革的可行性和成本，探索更强的安全保障机制。
Plain Language Accessible to non-experts
想象一个工厂，所有的指令和原料都存放在同一个仓库里。工人们根据这些指令生产产品，但如果有人偷偷把恶意的原料混进仓库，工厂就可能生产出有害的产品。这个工厂的设计没有专门的门或隔断来区分安全的指令和潜在的危险原料，所以任何人都可以把坏东西混进去，工厂就无法完全防止。这就像大模型中的指令和用户输入在同一空间，没有严格隔离，导致攻击者可以利用这个漏洞。要解决这个问题，就像在工厂里设置专门的门和隔断，确保指令和原料永远分开，才能真正保证安全。
ELI14 Explained like you're 14
你可以把大模型想象成一个超级智能的厨房，所有的食材和厨师的指令都放在一起，没有专门的柜子或门来区分。有人可能偷偷把坏的食材混进去，厨师不知道，从而做出有害的菜。这就像模型里的指令和用户输入在同一个空间，没有隔离，坏人就可以利用这个漏洞。为了让厨房更安全，我们需要设计一个专门的门或隔断，把安全的指令和潜在的危险内容分开。这样，即使有人试图偷偷放坏东西，也无法影响到厨师的决定。这个研究告诉我们，只有在架构上实现严格的隔离，才能真正防止这些攻击。否则，就像没有门的厨房，坏人总能找到机会搞破坏。
Abstract
Prompt injection is the top security risk for LLM-integrated applications, yet every defense proposed so far has been broken. We prove this is not a coincidence: in shared-embedding architectures that lack enforced control-data separation, perfect prompt-injection prevention is mathematically impossible. We formalize prompted systems as Prompted Action Models whose outputs include control-authoritative actions: refusal decisions, tool authorization, policy routing, and memory writes. We define Semantic-Faithful Control (SFC), the property that such behavior depends only on the meaning of untrusted input, not on how it is encoded. We then prove SFC is unachievable within the shared pipeline, via three results: a provenance-recovery impossibility (shared representations make trusted and untrusted content statistically inseparable, bounded by total variation distance); control-path exposure (untrusted tokens enter control-relevant computation through the same attention value-aggregation that determines outputs); and a finite-coverage invariance gap (finite training cannot certify invariance over infinite semantic-equivalence classes). We ground each quantity in measurements on production tokenizers and models. The result is structural, not a gap in current defenses. It mirrors the code-data confusion in Von Neumann machines that gives rise to buffer overflows, a vulnerability class that took decades of layered defenses (DEP, Write-XOR-Execute, ASLR, stack canaries, and ultimately memory-safe languages) to contain, because no single mechanism sufficed. The implication is the same: prompt injection cannot be eliminated by better in-pipeline classification or alignment alone. It requires architectural separation of instruction and data channels. We identify the root cause and the class of solution it demands.