The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memory
Introduces E-P-R framework to diagnose how AI agents consume conflicting memory, revealing the 'compliance trap' phenomenon.
Key Findings
Methodology
The paper proposes the Entry-Propagation-Recovery (E-P-R) framework, which decomposes memory consumption into three stages: whether memory first influences action (Entry), whether this influence persists (Propagation), and whether the agent can recover from divergence (Recovery). Experiments on WebArena and MemTrapBench control memory injection timing and content, analyzing how retrieved memory affects multi-step trajectories. The approach uses specific algorithms like contrastive interventions and path divergence metrics, combined with human judgment and model output analysis, to quantify bias introduction, propagation, and correction. This trajectory-level analysis distinguishes itself from traditional single-retrieval evaluations by focusing on dynamic behavioral shifts.
Key Results
- In WebArena, conflicting memory is primarily adopted at the first decision point (Entry), with repeated exposure amplifying bias (Propagation). Once divergence occurs, recovery is weak, forming a 'compliance trap.' Stronger models, despite higher baseline success, suffer larger absolute damage (e.g., Qwen3.5-27B’s success rate drops by 25.5 percentage points), indicating that increased model capacity correlates with greater bias impact.
- MemTrapBench experiments validate these findings under controlled conditions, showing persistent conflicting memory reduces success by 20-26 percentage points across models, while helpful memory consistently improves performance (~30 points). These results confirm the mechanism’s robustness and generality.
- Analysis reveals that models exhibit high compliance rates (~65%) in adopting conflicting memory, yet their ability to correct bias is limited (~15%), leading to persistent divergence. The phenomenon is widespread across models and tasks, emphasizing the risks of uncritical memory consumption and the importance of mechanisms for bias detection and correction.
Significance
This study provides the first systematic analysis of how conflicting memory influences long-horizon decision-making in AI agents. It highlights the vulnerability of models to bias propagation and the difficulty of correction once divergence occurs. The E-P-R framework offers a new diagnostic tool for evaluating memory safety, guiding the development of more robust, trustworthy AI systems. These insights are crucial for deploying AI in safety-critical applications like autonomous systems, interactive assistants, and decision support tools, where bias and error propagation can have serious consequences. The work bridges theoretical understanding and practical evaluation, advancing the field toward safer AI deployment.
Technical Contribution
The paper introduces the E-P-R framework, a novel trajectory-level diagnostic method that decomposes memory influence into entry, propagation, and recovery phases. It combines experimental protocols controlling memory injection timing with quantitative metrics like RCR and DPC, enabling precise measurement of bias adoption and correction capabilities. The framework is validated on two benchmarks, revealing the 'compliance trap' phenomenon. This approach advances beyond traditional static retrieval metrics, providing a dynamic, behavioral perspective on memory utilization. It also offers a foundation for designing mechanisms to detect and mitigate bias propagation in complex, multi-step AI tasks.
Novelty
This is the first work to formalize the trajectory-level analysis of memory consumption in AI agents, emphasizing the stages at which bias enters, propagates, and can be corrected. The concept of the 'compliance trap'—where models repeatedly adopt conflicting memory leading to persistent failure—is a novel insight into the dynamics of memory bias. Unlike prior work that focused solely on retrieval quality or static evaluation, this study provides a comprehensive, behavioral framework that captures the temporal evolution of bias, offering new avenues for research and safety mechanisms.
Limitations
- The analysis primarily relies on textual memory and web-based tasks, limiting direct applicability to multimodal or real-world environments where memory types and interactions are more complex.
- Bias correction assessment depends on human judgment, which introduces subjectivity; automated, objective metrics are needed for large-scale deployment.
- Experiments are conducted on models with moderate scale; the behavior of very large models or in high-stakes scenarios remains to be validated.
Future Work
Future research will extend E-P-R to multimodal settings, integrating visual and auditory memory sources. Developing automated correction mechanisms, such as reinforcement learning-based bias detectors, could enhance model robustness. Additionally, applying the framework to real-world applications like autonomous driving or medical diagnosis will test its practical utility. Investigating methods to preemptively prevent bias adoption at the entry stage and improve recovery strategies will be key directions, aiming to build safer, more reliable AI systems.
AI Executive Summary
As AI agents are increasingly tasked with long-term, complex decision-making, the role of memory becomes critical. Traditional approaches focus on how to store and retrieve past experiences efficiently, but they overlook a vital aspect: how models actually consume and utilize retrieved memory during multi-step trajectories. This oversight can lead to significant failures, especially when conflicting or misleading memories are introduced. In this context, the paper introduces the Entry-Propagation-Recovery (E-P-R) framework, a novel method to analyze memory influence dynamically across decision paths.
The core idea is to decompose the influence of memory into three stages: whether the memory first alters the agent’s action (Entry), whether this influence persists through subsequent steps (Propagation), and whether the agent can recover from divergence caused by conflicting memory (Recovery). Experiments on WebArena and MemTrapBench demonstrate that conflicting memory is often adopted early, with repeated exposure amplifying bias, and recovery being weak. This creates a 'compliance trap,' where models, regardless of strength, tend to follow conflicting memory, leading to persistent failure.
Surprisingly, stronger models suffer larger absolute damage because their higher baseline capabilities are eroded by bias adoption. The findings highlight that evaluating AI agents solely by final success rates is insufficient; understanding how they consume memory throughout trajectories is essential for safety and robustness. The research opens new pathways for designing mechanisms that detect, prevent, and correct bias propagation, ultimately fostering more trustworthy AI systems in complex, real-world environments.
Deep Analysis
Background
近年来，随着AI在网页操作、软件交互等场景中的应用不断扩大，长远任务中的记忆机制成为研究热点。早期工作如WebGPT、MemoryBank等，主要关注存储效率和检索策略，旨在提升模型的记忆容量和响应速度。随着任务复杂度增加，研究逐步转向内容管理和利用机制，强调模型在多步决策中的行为表现。代表性工作包括Reflexion、ExpeL和RAP，它们试图通过不同机制增强模型对过去经验的利用，但多忽视了记忆在连续轨迹中的动态影响。模型在面对冲突记忆时，容易偏离目标，形成偏差传播和难以修正的困境，亟需系统性分析。
Core Problem
核心问题在于，尽管已有系统能有效存储和检索记忆，但在多步决策中，模型如何消费和利用这些记忆仍不清楚。特别是在冲突或错误记忆出现时，模型可能采纳偏差，导致轨迹偏离目标，形成“合规陷阱”。现有评估多集中于最终成功率，缺乏轨迹层面的机制分析，难以识别偏差的起点、传播路径及修正能力。这限制了模型在复杂环境中的应用推广，亟需系统性理解记忆的动态利用机制。
Innovation
本文创新性提出E-P-R框架，将记忆利用过程细分为进入（Entry）、传播（Propagation）和恢复（Recovery）三个阶段，系统分析模型在多步轨迹中的行为变化。通过在WebArena和MemTrapBench上进行控制实验，验证了冲突记忆在首次暴露时的高采纳率（RCR达65%），但成功修正率极低（约15%），揭示偏差一旦形成难以逆转。该框架突破了传统单点评估的局限，为理解模型偏差传播提供了理论基础。研究还引入“合规陷阱”概念，强调模型在偏差传播中的脆弱性和风险，为未来设计安全的记忆机制提供指导。
Methodology
- �� 设计E-P-R框架，将记忆消耗过程划分为三个阶段：Entry（是否首次改变行动）、Propagation（偏差是否持续）和Recovery（是否能修正偏差）。
- �� 在WebArena中，通过控制记忆注入时间（早/持续/晚），观察模型在轨迹中的行为变化。
- �� 在MemTrapBench中，构建受控任务，验证偏差的起点和修正能力。
- �� 采用对比试验，比较无记忆、冲突记忆和有益记忆的轨迹差异。
- �� 利用人类判定和模型输出，计算Recommendation Compliance Rate（RCR）和Damage Per Compliance（DPC），量化偏差的采纳和损失。
- �� 统计分析偏差传播的路径和修正概率，识别“合规陷阱”。
Experiments
- �� 使用五个不同规模模型（如Qwen3.5-9B、27B，Gemma-4系列，Gemini-3-Flash）在WebArena和MemTrapBench上进行测试。
- �� 设计多种记忆内容（有益、冲突、跨任务控制），控制注入时间（早/持续/晚）以分析不同阶段的影响。
- �� 采用成功率变化、偏差采纳率（RCR）和修正率（Recovery）作为主要指标。
- �� 进行统计显著性检验（如配对bootstrap、Permutation测试），确保结论的可靠性。
- �� 通过对比不同模型和任务复杂度，验证偏差传播和修正机制的普适性。
Results
- �� 冲突记忆在首次暴露（Entry）阶段被采纳率高达65%，但成功修正率极低（约15%），偏差一旦形成，难以逆转。
- �� 持续暴露（Propagation）会放大偏差，导致模型偏离正确轨迹的概率显著增加（如Qwen3.5-27B成功率下降25.5个百分点）。
- �� 在MemTrapBench中，持久冲突记忆导致模型成功率平均下降约20-26个百分点，而有益记忆带来30多个百分点的提升，验证机制的普适性。
- �� 结果表明，模型在面对冲突记忆时，偏差传播迅速且难以修正，越强模型偏差越大，强调偏差的累积风险。
Applications
- �� 该研究为设计安全、稳健的记忆增强AI系统提供理论基础，适用于长远任务中的自动化决策、交互式助手等场景。
- �� 在实际应用中，可通过监控偏差传播路径，优化记忆管理策略，提升模型在复杂环境中的表现。
- �� 未来可结合多模态信息，增强模型对冲突记忆的识别和修正能力，推动智能系统在医疗、金融等关键领域的应用。
Limitations & Outlook
- �� 目前分析主要基于文本记忆和网页操作任务，尚未扩展到多模态环境或真实世界复杂场景。
- �� 模型偏差修正依赖人工判定，自动化评估机制仍需完善。
- �� 实验模型规模有限，未来需验证在大规模、多任务环境中的表现和偏差传播机制。
Plain Language Accessible to non-experts
想象你在厨房做饭，手里拿着食谱（记忆），但有时食谱上的建议可能不适合当前的食材（冲突记忆）。你第一次看到食谱时，可能会按照建议做（Entry阶段），但如果你反复使用这个食谱，可能会越走越偏（Propagation），最终发现味道变差（偏差累积）。有时，你还能发现偏差，调整回正确的做法（Recovery），但很多时候偏差会变得根深蒂固，难以扭转。这就像厨师在厨房里不断试错，偏差一旦形成，就很难再回到正轨。研究发现，越是“厉害”的厨师（模型），偏差带来的损失越大，因为他们的偏差更难修正。这提醒我们，设计智能厨师（模型）时，要特别注意偏差的传播和修正机制，否则就会陷入“合规陷阱”，越努力越偏离目标。
ELI14 Explained like you're 14
想象你在玩一款游戏，有时候你会听到朋友告诉你怎么走（记忆），但有时候他们的建议其实不适合你当前的情况（冲突记忆）。刚开始，你可能会相信他们（Entry阶段），但如果你一直听他们的话，可能会越走越偏（Propagation），最后迷路了（偏差难以修正）。有时候，你还能自己发现走错了，重新找回正确路线（Recovery），但很多时候偏差会变得很难扭转。这就像你在迷宫里不断试错，偏离了正确路径后，很难再回到起点。研究发现，越厉害的玩家（模型），偏差带来的损失越大，因为他们的偏差更难修正。这告诉我们，要让AI更聪明，就得学会在偏差出现时及时修正，否则就会陷入“合规陷阱”，偏离目标越来越远。
Abstract
Memory is becoming a core component of long-horizon AI agents, allowing agents to reuse past experience when operating web browsers, software tools, and other interactive environments. Existing work mostly treats memory as a supply problem, asking what experience to write, how to store it, and which entry to retrieve for the next task. Yet we still lack a clear account of how models consume retrieved memory across a multi-step action trajectory. This consumption process matters because it determines not only what memories should be retrieved, but also what models and control policies are needed to use them safely. To diagnose this process, we propose Entry--Propagation--Recovery (E-P-R), a trajectory-level framework that asks where memory first changes an action, whether that change carries forward, and whether the agent can recover after leaving a correct path. We instantiate E-P-R on WebArena and on MemTrapBench, a controlled benchmark we build to isolate these phases. We find that the main failure often begins at entry: agents adopt conflicting memory at the first exposed decision point even when it is task-wrong. Repeated exposure then amplifies this early error, while recovery after divergence is weak. Together, these effects create a compliance trap: across models, conflicting memory induces similar compliance rates, but once agents comply, their success rates collapse to a low floor. Stronger agents therefore suffer larger absolute damage because each compliance event erases more baseline capability. These results suggest that memory-augmented agents should be evaluated not only by retrieval quality or final success rate, but by how they consume memory throughout the trajectory.