LatentPress: Context Compression Beyond Text and Vision
LatentPress encodes long contexts into continuous soft tokens with 4-16× compression, enabling direct reading without text reconstruction, boosting efficiency.
Key Findings
Methodology
LatentPress introduces a soft-token interface where a frozen decoder directly reads compressed conversational histories and long documents encoded as continuous vectors. It employs a lightweight, role-aware compression strategy, dynamically allocating compression rates based on input structure, trained via a loss combining reconstruction and KL divergence. Only a small adapter is trained, leaving the decoder frozen, enabling 4-16× compression without text reconstruction. Experiments on LongMemEval and LongBench-QA demonstrate that the compressed representations preserve or surpass raw context accuracy, with significantly faster inference and writing speeds.
Key Results
- On LongMemEval, LatentPress achieves 0.504 accuracy at 7.70× compression, outperforming uncompressed evidence (0.490), text summaries (0.184), and OCR-based compression (0.426→0.312).
- In LongBench-QA, in-domain trained writers match or exceed raw context performance at 4-8× compression, with 16× trailing slightly behind. Writing takes 43ms per conversation, over ten times faster than traditional summarization or OCR.
- Across different model sizes (Qwen2.5-7B, 8B, 14B), LatentPress maintains strong transferability, consistently outperforming baseline compression methods in accuracy and speed.
Significance
This work advances the state-of-the-art in long context compression by enabling direct, efficient reading via soft tokens, bypassing text reconstruction. It addresses key bottlenecks in large-scale NLP systems, facilitating faster, more scalable long-text and multi-turn dialogue understanding. Its minimal training overhead and high compression ratio make it practical for deployment in real-time applications, such as chatbots, knowledge bases, and multimodal systems, pushing the boundaries of large model capabilities.
Technical Contribution
LatentPress's main innovation lies in the direct read interface using soft tokens, trained with a tiny adapter, and a role-aware compression scheme. Unlike prior methods like Gist or AutoCompressor, it does not rely on autoencoding or retriever-based approaches, instead operating entirely in the embedding space. Its structure allows variable compression rates across segments, optimizing information retention. This approach significantly reduces training and inference costs while maintaining high accuracy, opening new avenues for scalable long-context modeling.
Novelty
This is the first approach to combine a frozen decoder with a trainable, role-aware soft-token encoder that compresses multi-turn dialogues and long documents without text reconstruction. Its dynamic, structure-informed compression strategy and direct input of soft tokens into the frozen decoder represent a novel paradigm in long context modeling, surpassing prior autoencoder or retrieval-based methods in efficiency and flexibility.
Limitations
- The compression strategy relies on manually designed role-aware heuristics, which may not generalize well across different content types or tasks. Automated, learned compression policies are needed for broader applicability.
- At very high compression ratios (e.g., 16×), some information loss affects complex reasoning and knowledge updating, limiting performance in certain scenarios.
- The current approach is primarily tested on static, oracle evidence, and its effectiveness with dynamic retrieval or multi-modal inputs remains to be validated.
Future Work
Future directions include developing reinforcement learning-based adaptive compression policies, enabling the system to automatically allocate compression based on content importance. Extending the framework to multi-modal data, such as images and tools, and integrating with retrieval modules for dynamic long-term memory are promising avenues. Additionally, exploring multi-task training and broader generalization across domains will further enhance its robustness and utility.
AI Executive Summary
Long conversations and extensive documents are fundamental to advanced AI applications, yet traditional approaches struggle with efficiency and scalability. Text summaries and OCR-based reconstructions, while useful, introduce latency and potential information loss. LatentPress offers a transformative solution by encoding long contexts into continuous soft tokens that a frozen decoder can read directly, bypassing text reconstruction. This method achieves 4-16× compression ratios, drastically reducing storage and inference time.
The core innovation involves training a lightweight adapter to map segments of dialogue or document content into soft tokens within the decoder’s embedding space. These tokens are dynamically allocated based on input structure, such as roles in conversations, ensuring critical information is preserved. During inference, the system reads these soft tokens directly, enabling near real-time responses with minimal latency. Experimental results on benchmarks like LongMemEval and LongBench-QA demonstrate that LatentPress maintains or exceeds the accuracy of uncompressed contexts, with inference speeds 5-9× faster.
This approach addresses longstanding challenges in long-text processing, offering a scalable, efficient, and flexible framework for large language models. Its minimal training overhead and high compression ratio make it suitable for deployment in real-world applications like chatbots, knowledge management, and multimodal systems. The ability to operate without text reconstruction opens new possibilities for multi-turn dialogue and multi-modal content management, paving the way for more responsive and resource-efficient AI systems.
Looking ahead, the authors plan to develop adaptive, learned compression policies and extend the framework to multi-modal data, further enhancing its applicability. Overall, LatentPress marks a significant step toward scalable, high-performance long context understanding in AI.
Deep Analysis
Background
近年来，深度学习模型在自然语言处理中的表现持续提升，尤其在长文本理解和多轮对话方面取得显著进展。早期方法主要依赖文本摘要、检索或视觉重建技术，存在信息损失和处理延迟的问题。Gist、AutoCompressor、ICAE等方法尝试将上下文编码为连续向量，但多依赖重建或检索机制，限制了效率和适应性。随着模型规模的扩大，长文本存储和处理成为瓶颈，亟需更高效的压缩和读取机制。近年来，软标记（soft tokens）作为一种潜在解决方案逐渐受到关注，旨在在保持信息完整的同时减少存储和计算成本。本论文在此基础上提出LatentPress，结合结构化压缩和角色感知策略，推动长文本和多轮对话的高效连续记忆技术。
Core Problem
长文本和多轮对话的存储与理解面临巨大挑战。传统方法依赖文本摘要或OCR重建，存在信息丢失、处理慢、成本高的问题。如何在保证信息完整的前提下，实现更高压缩比和更快的推理速度，成为核心难题。尤其是在模型冻结的情况下，设计一种无需文本重建的直接读取接口，成为研究焦点。现有技术缺乏兼顾效率和信息保留的解决方案，限制了长文本和对话系统的实用性和扩展性。这一问题的解决，将极大推动智能系统在实时性和规模上的突破。
Innovation
本研究的创新点主要包括：1）引入连续空间中的软标记作为长文本和对话的压缩载体，避免文本重建，显著提升效率；2）设计角色感知的压缩策略，根据对话角色动态调整每段的压缩率，增强信息保留；3）仅训练少量适配器，保持解码器冻结，降低训练成本。相比以往依赖重建或检索的方案，LatentPress在保持模型性能的同时，大幅减少了存储和推理时间。其结构化压缩机制结合输入内容的结构信息，实现了高效的多轮对话和长文理解，为大模型长文本应用提供了新思路。
Methodology
- �� 将长文本或对话拆分为多个段落或轮次，作为输入序列；
- �� 设计角色感知的压缩策略，根据角色类型动态调整每段的压缩率；
- �� 训练一个轻量级的适配器，将每段内容映射为连续软标记，输入到冻结的解码器中；
- �� 采用目标函数结合重建损失和KL散度，确保编码信息的完整性；
- �� 在推理阶段，直接用软标记作为输入，无需文本重建；
- �� 通过不同的压缩比例，平衡信息保留和存储效率，实现4-16倍压缩；
- �� 在LongMemEval和LongBench-QA任务中测试模型的准确性和效率，验证其迁移和泛化能力。
Experiments
采用LongMemEval和LongBench-QA两个公开数据集，评估压缩效果和推理速度。对比未压缩、文本摘要、OCR和不同压缩率的LatentPress，使用Qwen系列模型作为基础解码器。指标包括准确率、压缩比、写入时间和推理延迟。实验设计涵盖跨域和内域迁移，验证模型在不同场景下的表现。超参数包括压缩比例（4-16倍）和角色感知策略，进行多轮消融分析，确保方法的稳健性和适应性。
Results
LatentPress在LongMemEval中，7.70倍压缩率下达成0.504准确率，优于未压缩的0.490，且明显优于文本摘要（0.184）和OCR（0.426→0.312）。在LongBench-QA中，4-8倍压缩下，压缩后效果与原始持平或更优，16倍略低。写入速度仅需43毫秒，远快于传统方法的几百毫秒。模型在不同规模（Qwen2.5-7B、8B、14B）上表现一致，验证了其迁移和泛化能力。这些结果表明，LatentPress在保持高准确率的同时，大幅提升了长文本和对话的处理效率。
Applications
该技术适用于智能客服、知识库管理、多轮对话系统和多模态信息融合场景。只需在模型端引入软标记编码，无需复杂的检索或重建流程，即可实现高效存储和快速响应。未来可结合动态压缩策略，适应不同任务和内容类型，推动大规模长文本理解的实用化。长远来看，该方案有望成为大模型在实时交互和大规模知识管理中的核心技术之一。
Limitations & Outlook
当前方法依赖手工设计的角色感知和压缩策略，缺乏自动优化机制，可能在内容结构复杂或变化频繁的场景中表现不佳。高压缩比（如16倍）会导致信息丢失，影响推理和知识更新能力。模型在特定任务和数据集上训练，泛化到多模态或动态环境仍存在挑战。未来需引入自适应策略和多模态编码，提升鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在整理一大堆书和笔记，要让别人快速理解内容，你可以用简短的标签或关键词来代表每本书的核心信息，而不是逐字逐句地复述。LatentPress就像用这些关键词标签，把长长的对话或文章变成一串简洁的符号，这些符号可以直接告诉模型“这段话讲的是什么”，而不用重新写一遍。这样一来，存储和传输都变得更快更轻松，就像用便签代替整本书一样。模型只需要看这些标签，就能快速理解和回答问题，效率大大提升。这种方法特别适合处理海量信息，比如聊天记录、长篇文章或多模态内容，节省了大量时间和计算资源。
ELI14 Explained like you're 14
你知道我们平时写笔记的时候，会用一些简短的关键词或者符号来代表一大段内容吗？比如“考试”、“朋友”、“电影”，这样就不用每次都写一大堆字。LatentPress就像这样，把长长的对话或者文章变成一串特别短的符号（叫软标记），模型可以直接看这些符号，知道它们代表的意思，然后快速回答问题。这样一来，存储信息变得很快，模型反应也更快，就像用便签代替整本书一样。它还能根据内容的不同角色（比如用户还是助手）调整压缩的程度，既节省空间，又不丢失重要信息。这个方法让长对话和长文章的理解变得更高效，特别适合用在聊天机器人、知识库和多模态系统中。
Abstract
Compressed context is usually carried as human-readable text or as rendered images that must be decoded, even when its consumer is a language model. We introduce LatentPress, which writes conversational histories and long documents into a third representation: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference. A small reader-matched writer compresses $4$-$16\times$ while training only an adapter (4.2M-26.2M parameters, $\sim\!0.1\%$ of the decoder). On LongMemEval, LatentPress reaches $0.504$ accuracy at $7.70\times$ compression versus $0.490$ for uncompressed evidence, outperforming text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at $4$-$8\times$ compression, while $16\times$ trails raw. Writing takes 43ms per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is $5$-$9\times$ faster than raw context or cached OCR. We validate the interface under two transfer settings, zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived QA to unseen LongBench document domains, establishing direct soft tokens as a practical machine-facing context interface beyond text and vision. The implementation of the experiments could be found at: https://github.com/xuyd16ai/context_softtoken_compress .