Reliable Post-Retrieval Assembly for Agent Memory: Separating Evidence Extraction from Policy Execution
Proposes structured post-retrieval assembly, boosting memory accuracy to 82%/93% single-hop and 27%/41% multi-hop, surpassing all prior results.
Key Findings
Methodology
This work introduces a two-stage process: first, an LLM extracts semantically matching evidence into a structured candidate set; second, a separate policy executor applies explicit rules, such as selecting the newest version based on metadata. The evidence extraction uses prompts to identify all relevant candidates without immediate decision-making, reducing cognitive load. The policy stage then independently enforces rules, avoiding entanglement of semantic filtering, conflict resolution, and answer generation. Experiments on the FactConsolidation benchmark with BM25 retrieval and GPT-4 models demonstrate that this separation achieves 82%/93% single-hop and 27%/41% multi-hop accuracy at 262K data scale, outperforming all existing systems. Controlled ablation studies confirm that most gains stem from decoupling evidence identification from final policy execution, rather than model inference improvements.
Key Results
- At 262K, the structured assembly pipeline reaches 82%/93% single-hop and 27%/41% multi-hop accuracy, exceeding all previously reported results on the MAB v3 FactConsolidation task, including long-context and graph-based methods.
- Comparison with direct-answer approaches shows a 10.8 percentage point average improvement at 262K, with the gap widening to 21 points at larger scales, mainly due to the explicit evidence structuring.
- Isolating the policy executor reveals only a 2-point gain, indicating that the main benefit arises from the evidence extraction and structuring step, not from the policy execution method itself.
Significance
This research highlights the importance of a modular approach in memory systems, where separating evidence retrieval, structuring, and policy enforcement enhances reliability and interpretability. It addresses fundamental challenges in dynamic knowledge management, conflict resolution, and multi-hop reasoning, paving the way for more trustworthy AI agents capable of handling evolving information. The findings suggest that architectural design choices—particularly evidence structuring and explicit version control—are crucial for scaling up dependable lifelong memory systems, with broad implications for AI deployment in real-world scenarios involving continual learning and knowledge updates.
Technical Contribution
The paper introduces a formal framework for post-retrieval evidence assembly, emphasizing the decoupling of semantic evidence extraction from explicit policy enforcement. It demonstrates that structured intermediate representations, combined with version metadata, significantly improve multi-hop reasoning accuracy. The approach leverages a two-stage process—semantic candidate extraction followed by rule-based selection—enabling independent testing, logging, and enforcement of policies. Empirical results on the FactConsolidation benchmark show that this architecture outperforms existing end-to-end models, establishing a new paradigm for reliable memory in large language models.
Novelty
This work is the first to formalize and empirically validate the separation of evidence identification from policy execution in large-scale memory systems. Unlike prior end-to-end approaches, it explicitly structures evidence with version metadata, enabling robust conflict resolution. The innovation lies in the modular design that improves accuracy and interpretability, especially in multi-hop reasoning tasks, setting a new standard for scalable, reliable memory architectures.
Limitations
- The current approach relies heavily on explicit version metadata, limiting applicability to scenarios with well-defined update rules. It struggles with partial orders, causal dependencies, and complex temporal reasoning.
- Most errors originate from the initial evidence extraction stage, especially in noisy or ambiguous data, indicating room for improvement in candidate retrieval robustness.
- Experiments are primarily conducted on synthetic, benchmark datasets; real-world data with high noise or multiple conflicting sources may pose additional challenges.
Future Work
Future directions include integrating causal and temporal reasoning mechanisms, handling partial orders, and extending the framework to more complex knowledge update scenarios. Incorporating graph-based structures and causal inference models could further enhance conflict resolution. Additionally, scaling the approach to real-world, noisy datasets and multi-source environments will be crucial for deploying reliable lifelong memory agents in practical applications.
AI Executive Summary
Building reliable long-term memory systems for AI agents remains a central challenge, especially in dynamic environments where facts evolve and conflicts arise. Traditional end-to-end models often entangle evidence retrieval, conflict resolution, and answer generation, leading to brittleness and unreliability. This study introduces a structured post-retrieval assembly framework that explicitly separates evidence extraction from policy execution, forming a modular architecture that enhances robustness and interpretability.
The core idea is to first convert retrieved evidence into a structured candidate set using an LLM, which identifies all relevant facts without making immediate decisions. Then, a dedicated policy executor applies explicit rules—such as selecting the most recent version based on metadata—to determine the final answer. This decoupling reduces the cognitive load on the model, minimizes error propagation, and allows independent testing and enforcement of policies.
Experimental results on the FactConsolidation benchmark with BM25 retrieval and GPT-4 models demonstrate that this approach achieves 82%/93% accuracy for single-hop questions and 27%/41% for multi-hop questions at a scale of 262K data points, surpassing all prior published systems. Ablation studies confirm that most performance gains come from the evidence structuring step, not just model inference improvements. The findings underscore that the key to reliable memory is not merely sophisticated storage but effective evidence assembly and conflict resolution.
While promising, the method currently relies on explicit version metadata, limiting its scope to current-value conflicts. Future work aims to incorporate causal reasoning, handle complex temporal dependencies, and extend to noisy, real-world data. Overall, this research advances the design of trustworthy, scalable lifelong memory systems, crucial for deploying AI in dynamic, knowledge-rich environments.
Deep Dive
Abstract
LLM-based memory systems can retrieve relevant evidence yet still fail when answer generation entangles semantic filtering, conflict resolution, prior suppression, and output generation in one step. We study this failure as a problem of post-retrieval assembly. In the MemoryAgentBench (MAB) release used here, FactConsolidation explicitly states that newer facts have larger serial numbers, yet the best reported retrieval/memory result is 54% single-hop and all 22 reported systems score at most 7% multi-hop. We evaluate a structured assembly interface in which an LLM first extracts semantically matching evidence into a candidate representation and a separate stage executes the required answer policy. At 262K, this pipeline reaches 82%/93% single-hop and 27%/41% multi-hop with gpt-4o-mini/gpt-4o, exceeding every result reported in the MAB v3 FactConsolidation comparison. This is a task-level result, not a claim that the evaluated memory architectures are broadly inferior. A controlled whole-pipeline comparison, with identical backbone, retrieved top-10 evidence, chunking, and n=100 per cell, improves single-hop accuracy by 10.8 percentage points (pp) on average and 21 pp at 262K. A targeted comparison using the same extraction setup shows that changing only the final policy executor contributes 2.0 pp on average and 0 pp at 262K. Most of the gain therefore comes from separating evidence identification from final policy execution rather than from the freshness operator itself. A LongMemEval check finds no significant overall advantage (26/45 versus 29/45; paired exact McNemar p=0.45), bounding the result to current-value questions with explicit version metadata. The evidence identifies post-retrieval assembly as a distinct reliability boundary between retrieval and answer generation.