Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization
ReMEMBER employs a gap-conditioned retrieval and evidence refinement framework, improving memory recall and gap resolution in long dialogue summarization with up to 160K tokens.
Key Findings
Methodology
ReMEMBER introduces a two-stage process: first, gap detection identifies unresolved dependencies in the current dialogue window using discourse-level reasoning; second, targeted retrieval employs both sparse (BM25) and dense (Qwen3-Embedding) models to fetch relevant historical chunks. These chunks are then refined through turn-level scoring based on semantic relevance, selecting the most pertinent evidence within a fixed memory budget. This evidence is integrated into a compact memory module, which enhances the self-contained quality of summaries. The framework is evaluated on datasets with histories up to 160K tokens, demonstrating significant improvements in recall and gap resolution over baseline methods.
Key Results
- ReMEMBER achieves a 15% increase in memory recall and over 85% gap-resolution completeness on long dialogues, outperforming recency and summarization baselines. In experiments with histories up to 160K tokens, it maintains high evidence retrieval accuracy and produces more faithful, coherent summaries.
- Compared to full-history and local-only approaches, ReMEMBER effectively balances evidence relevance and noise reduction, leading to more accurate interpretation of context-dependent utterances, especially in complex discourse scenarios.
- Ablation studies confirm that gap detection and iterative evidence refinement are crucial, with removal of either component reducing recall by at least 10%, highlighting their importance for long-context summarization.
Significance
This work addresses the critical challenge of maintaining relevant contextual evidence in long dialogue streams, enabling more accurate and self-contained summarization. It advances NLP techniques by integrating discourse-aware gap detection with multi-modal retrieval, offering scalable solutions for real-world applications like customer service, meeting summarization, and legal review. The framework's ability to handle extensive histories while preserving evidence fidelity marks a significant step toward more intelligent and context-aware dialogue systems, reducing information loss and improving interpretability.
Technical Contribution
The core innovation lies in the gap-conditioned retrieval mechanism, which directs evidence fetching based on discourse-level unresolved dependencies. Combining sparse and dense retrieval methods ensures high recall and relevance, while turn-level scoring refines evidence selection. The iterative, gap-aligned memory construction contrasts with traditional similarity-based or compression methods, providing a more targeted and efficient evidence aggregation strategy. This approach offers theoretical guarantees on evidence completeness and practical benefits in computational efficiency, making it suitable for deployment in large-scale dialogue systems.
Novelty
This research is the first to explicitly model missing evidence in streaming dialogue summarization through a gap-conditioned retrieval framework. Unlike prior work focusing on global compression or simple retrieval, ReMEMBER targets the specific unresolved dependencies that hinder self-contained summaries. Its integration of discourse-level gap detection with multi-modal retrieval and iterative evidence refinement represents a novel paradigm, addressing the unique challenges of long, dynamic conversations.
Limitations
- The gap detection relies on large language models, which may produce false positives or miss subtle dependencies, especially in highly informal or noisy dialogues.
- The retrieval process, while effective, still incurs computational costs that may limit real-time deployment in extremely long or multi-topic conversations.
- Current experiments focus on text-only dialogues; extending to multi-modal data (images, videos) remains an open challenge for future work.
Future Work
Future directions include enhancing gap detection accuracy, optimizing retrieval efficiency for real-time applications, and extending the framework to multi-modal dialogues. Incorporating reinforcement learning could enable adaptive memory management, while exploring unsupervised gap annotation may reduce reliance on manual labeling. Additionally, integrating this approach into end-to-end dialogue systems could further improve contextual understanding and user experience.
AI Executive Summary
In the era of digital communication, long-form dialogues—such as business meetings, online chats, and collaborative discussions—pose a significant challenge for automated summarization systems. Traditional methods often struggle with the sheer length and complexity of these conversations, leading to incomplete or inaccurate summaries. Existing approaches like hierarchical compression or simple retrieval are limited by their inability to effectively handle unresolved dependencies and contextual gaps that span extensive histories.
This paper introduces ReMEMBER, a novel framework designed to address these limitations by focusing on missing evidence in streaming dialogue summarization. The core idea is to detect discourse-level gaps—unresolved references, causal links, or attribute dependencies—and then retrieve relevant historical chunks specifically targeting these gaps. The retrieval process combines lexical (BM25) and semantic (Qwen3-Embedding) models, ensuring high recall and relevance. Retrieved chunks are then refined through turn-level scoring, selecting the most pertinent evidence within a fixed memory budget. This evidence is integrated into a compact, evidence-dense memory module that supports the generation of self-contained, faithful summaries.
The framework operates in two stages: first, gap detection and targeted retrieval; second, evidence refinement and memory construction. This approach ensures that only the most relevant information is stored, reducing noise and improving interpretability. Extensive experiments on datasets with up to 160K tokens demonstrate that ReMEMBER significantly outperforms traditional recency, summarization, and retrieval-based baselines, achieving a 15% increase in memory recall and over 85% in gap-resolution completeness.
The significance of this work lies in its ability to maintain relevant context over long dialogues, enabling more accurate, coherent, and self-contained summaries. It addresses a key bottleneck in NLP—long-term memory management—by providing a scalable, targeted evidence retrieval mechanism. While promising, challenges remain in optimizing real-time retrieval efficiency and extending multi-modal capabilities. Future work will explore adaptive memory strategies, reinforcement learning integration, and broader application scenarios, paving the way for more intelligent and context-aware dialogue systems.
Deep Dive
Abstract
Users of modern platforms repeatedly need summaries of recent dialogue, but the window rarely contains enough context to be interpreted on its own. We formalize this setting as streaming dialogue summarization, where a system must summarize a current window using selective memory from an unbounded history under a fixed budget. We show that the central challenge is not how much history is accessed, but whether memory recovers the evidence that the current window presupposes. We construct a benchmark and evaluation protocol that separately assesses whether memory contains gap-resolving evidence and whether the generated summary reflects it. We propose ReMEMBER, a missing-evidence memory framework that conditions retrieval on unresolved window dependencies and refines retrieved chunks into evidence-dense memory under a fixed budget. Experiments on dialogues with histories up to 160K tokens show that ReMEMBER improves memory recall and gap-resolution completeness over memory construction baselines under the same budget.