Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents
Proposes in-loop memory for language agents, reducing latency to 100μs and improving task efficiency and accuracy.
Key Findings
Methodology
The paper introduces 'memory in the loop,' moving memory from external networks to in-process storage. This reduces latency to ~100μs, enabling memory access at every reasoning step. Experiments validate its impact on task performance.
Key Results
- Task recall in GPT-5 models improved from 0/5 to 3.6-4.8/5 with in-loop memory, achieving 80-165μs latency.
- Redundant actions dropped to 0/12 with in-loop memory, compared to 7.2/12 for 110ms network storage latency.
- Embedding latency reduced from 200-400ms to 40μs using a local embedder, significantly boosting overall performance.
Significance
This work redefines memory in language agents, transforming it from an external tool to an integral part of reasoning. It addresses the bottleneck of high-latency storage, paving the way for low-latency cognitive systems.
Technical Contribution
Demonstrates the causal relationship between memory latency and reasoning efficiency. Proposes a latency budget design criterion and validates the effectiveness of in-loop memory through experiments.
Novelty
First to apply the parity principle from extended cognition theory to language agents, introducing a novel reasoning loop architecture centered on low-latency memory.
Limitations
- The read policy limits full utilization of memory, causing some recall failures.
- Experiments are limited to GPT-5 models, raising questions about generalizability.
- Performance in large-scale distributed environments remains unexplored.
Future Work
Future work could explore smarter read policies, adaptation to distributed systems, and better integration of memory with local embedders.
AI Executive Summary
Language agents typically treat memory as an external tool, queried once per turn. This design is constrained by high-latency network storage, limiting task efficiency and accuracy. This paper proposes a novel architecture where memory is integrated into the reasoning loop as extended working memory.
By moving memory from external networks to in-process storage, latency is reduced from 110ms to ~100μs, enabling efficient memory access at every reasoning step. Experiments show task recall in GPT-5 models improves from 0/5 to 3.6-4.8/5, with redundant actions dropping to 0/12. A local embedder further reduces embedding latency to 40μs, optimizing overall performance.
This work redefines memory's role in language agents, transforming it into a core component of reasoning. While challenges remain, such as optimizing read policies and scaling to distributed systems, this approach lays the groundwork for future low-latency intelligent systems.
Deep Analysis
Background
Language agents operate in an observe-reason-act loop, but memory is treated as an external database queried once per turn. This design suffers from high-latency network storage (50-200ms), forcing systems to limit memory access frequency. Recent work focuses on optimizing access frequency but does not challenge the assumption of external memory.
Core Problem
The core issue is that existing memory architectures cannot support high-frequency access, leading to inefficient reasoning. Network storage latency forces systems to ration memory queries, making them unsuitable for complex tasks requiring frequent memory access.
Innovation
The paper proposes moving memory from external networks to in-process storage, reducing latency to ~100μs. By applying the parity principle from extended cognition theory, the authors redefine memory as an integral part of the reasoning process.
Methodology
- �� Introduced 'memory in the loop,' integrating memory into the reasoning process.
- �� Used a local embedder to reduce embedding latency from 200-400ms to 40μs.
- �� Designed experiments to validate the impact of low-latency memory on task performance.
- �� Analyzed the causal relationship between memory latency and reasoning efficiency, proposing a latency budget design criterion.
Experiments
Experiments used GPT-5 models in a six-turn trip-planning task where users provided five constraints early on, later asking the agent to recall them. A window limit (w=4) ensured early information scrolled out of view. Conditions compared no memory, in-loop memory, and gated memory strategies.
Results
In-loop memory improved task recall from 0/5 to 3.6-4.8/5, with redundant actions reduced to 0/12. Using a local embedder further reduced overall latency to 40μs, significantly enhancing performance.
Applications
This approach is applicable to tasks requiring high-frequency memory access, such as complex dialogue systems, real-time decision support, and intelligent search engines.
Limitations & Outlook
Current experiments focus on single-machine setups, leaving distributed performance untested. Additionally, optimizing read policies could further improve recall rates.
Plain Language Accessible to non-experts
Imagine you're cooking, and every time you need an ingredient, you have to run to a store down the street. That's like traditional language agents using network storage. This paper's idea is like having all your ingredients right in your kitchen — quick and convenient, making cooking much faster and easier.
ELI14 Explained like you're 14
Think of playing a video game where you have to pause and go to a storage room every time you need an item. Annoying, right? Now imagine your character has a backpack with everything you need. That's what this paper is about — giving AI its own 'backpack' of memory to make it super fast and smart!
Glossary
Extended Cognition
A theory suggesting external tools can become part of cognitive processes.
Used to justify in-process memory as part of reasoning.
Memory in the Loop
A design where memory is accessed at every reasoning step.
Core concept enabling efficiency improvements.
Parity Principle
A criterion from extended cognition theory requiring tools to be fast and easy to use.
Evaluates memory's role in reasoning.
Local Embedder
A technique reducing embedding latency to 40μs.
Optimizes memory loop performance.
Redundant Actions
Repeated actions already completed by the agent.
Metric for evaluating memory loop efficiency.
Open Questions Unanswered questions from this research
- 1 How can low-latency memory be implemented in distributed environments?
- 2 Can smarter read policies further improve recall rates?
- 3 What is the applicability of low-latency memory to other model architectures?
Applications
Immediate Applications
Intelligent Dialogue Systems
Enhances context understanding for chatbots and virtual assistants in customer service.
Real-Time Decision Support
Supports real-time complex decisions in fields like healthcare and finance.
Long-term Vision
General AI
Lays the foundation for developing human-level cognitive systems.
Abstract
Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.