Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction
Introduced COMPINT suite to address session constraint loss in context compaction, achieving 90% retention.
Key Findings
Methodology
The paper introduces the COMPINT evaluation suite to assess the retention of session constraints during context compaction. By implementing an SC-aware extractor running alongside compactors, it significantly improves constraint retention.
Key Results
- Current compactors retain only 17% of session constraints on average, while the SC-aware extractor achieves over 90% retention across scenarios.
- Retention varies sharply with compactor choice, prompt, context length, SC phrasing, and injection location.
- COMPINT suite validated the effectiveness of the SC-aware extractor.
Significance
This research addresses the loss of session constraints during context compaction, proposing a plug-and-play module that significantly enhances retention. It is crucial for improving task integrity in LLM systems.
Technical Contribution
The paper proposes a novel SC-aware extractor compatible with existing compactors, enhancing constraint retention without modifying LLM or compactors.
Novelty
This is the first systematic study on the impact of context compaction on session constraints, providing an effective solution.
Limitations
- While the SC-aware extractor improves retention, constraints may still be lost in complex scenarios.
- The study does not address the impact of different LLM architectures on SC retention.
Future Work
Future research could explore the impact of different LLM architectures on SC retention and optimize the SC-aware extractor for more scenarios.
AI Executive Summary
In long-context scenarios, LLM systems often need to compact context to continue tasks. However, this compaction process can lead to the loss of user-issued session constraints, affecting task integrity. To address this issue, the paper introduces the COMPINT evaluation suite, specifically designed to assess the retention of session constraints during context compaction.
The study finds that current compactors retain only 17% of session constraints on average, performing worse than running the same task without compaction. To tackle this, the authors propose an SC-aware extractor that runs alongside compactors as a plug-and-play module, achieving over 90% retention without modifying the LLM or compactors.
This research not only reveals the systematic impact of context compaction on session constraints but also provides an effective solution, significantly enhancing task integrity in LLM systems. Future research could further explore the impact of different LLM architectures on SC retention and optimize the SC-aware extractor for more scenarios.
Deep Analysis
Background
With the advancement of LLM technology, long-context processing capabilities have significantly improved. However, existing context compaction methods often overlook user-issued session constraints in complex tasks, leading to compromised task integrity. This paper proposes a new evaluation suite to address this issue.
Core Problem
Session constraints are lost during context compaction, affecting task integrity. Existing compactors fail to effectively retain these constraints, leading to ignored user instructions.
Innovation
Introduced COMPINT evaluation suite specifically for assessing the impact of context compaction on session constraints. Implemented an SC-aware extractor running alongside compactors, significantly improving constraint retention.
Methodology
- �� Introduced COMPINT evaluation suite to assess context compaction impact on session constraints.
- �� Proposed SC-aware extractor running alongside compactors as a plug-and-play module.
- �� Validated extractor effectiveness through experiments.
Experiments
Experimental design includes three long-context scenarios: multi-turn chat, agent trajectory, and long-horizon research. Compared different compactors to evaluate SC-aware extractor retention.
Results
Experiments show current compactors retain only 17% of session constraints, while SC-aware extractor achieves over 90% retention across scenarios.
Applications
The research can be applied to improve task integrity in LLM systems for complex tasks, suitable for long-context processing scenarios like customer service and automated research.
Limitations & Outlook
While the SC-aware extractor improves retention, constraints may still be lost in complex scenarios. Future research could optimize the extractor for more scenarios.
Plain Language Accessible to non-experts
Imagine working in a busy kitchen. You instruct your assistant not to discard any ingredients until you confirm, but during cleanup, the assistant forgets this directive, leading to the loss of important ingredients. This paper studies a similar issue and proposes a method to ensure the assistant remembers your instructions during cleanup.
ELI14 Explained like you're 14
Imagine playing a complex game where you tell your teammates not to take action until you confirm. But as the game gets complicated, they forget your instructions, leading to a game failure. This paper studies a similar issue and proposes a method to ensure teammates remember your instructions in complex situations.
Glossary
Context Compaction
The process of reducing context window pressure by summarizing conversation history to continue tasks.
Used in long-context scenarios to reduce memory burden.
Session Constraints
User-issued instructions to constrain LLM behavior within the current session.
Prone to loss during context compaction.
SC-aware Extractor
A module running alongside compactors to enhance session constraint retention.
Used as a plug-and-play module without modifying LLM or compactors.
COMPINT Evaluation Suite
A toolset for assessing the impact of context compaction on session constraints.
Validated through experiments.
Long-context Scenarios
Task scenarios requiring extensive information processing, such as multi-turn chat and long-horizon research.
Used to test compactor retention rates.
Open Questions Unanswered questions from this research
- 1 How to optimize the SC-aware extractor for more complex scenarios?
- 2 What is the impact of different LLM architectures on session constraint retention?
Applications
Immediate Applications
Customer Service
Enhance task integrity in LLM systems for customer service by ensuring accurate execution of user instructions.
Long-term Vision
Automated Research
Apply in automated research to ensure integrity of user instructions in long-term tasks.
Abstract
When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks. We identify a class of user-issued instructions, Session Constraints (SCs), such as "do not delete any emails until I confirm," that are meant to constrain LLM's behavior for the remainder of a session but are silently dropped during compaction. To quantify this loss, we introduce COMPINT, an evaluation suite that evaluates compactors across three long-context scenarios: multi-turn chat, agentic trajectory, and long-horizon research. Current compactors retain only 17% of injected SCs on average, and most perform worse than running the same task without compaction. Retention varies sharply with compactor, prompt, context length, SC phrasing, and injection location, showing that the loss is systematic rather than tied to any single setting. We propose an SC-aware extractor that runs alongside the compactor as a plug-and-play module, achieving over 90% retention across all three scenarios without modifying the compactor or LLM. The COMPINT evaluation suite and accompanying implementation are available at https://github.com/ZhiqiEliWang/compaction-integrity.