Long Context Compression with Activation Beacon
Activation Beacon introduces progressive activation compression, enabling efficient long-text processing with up to 8x compression, doubling inference speed.
Key Findings
Methodology
This paper proposes Activation Beacon, a plug-in module for transformer-based LLMs that employs beacon tokens to progressively compress activations (keys and values) at each layer. The approach segments long input sequences into chunks, interleaves beacon tokens, and encodes each chunk by distilling its information into beacon token activations. During training, the model uses compression-based auto-regression with randomly sampled compression ratios, supporting flexible configurations. The method reuses cached beacon activations to handle sequences exceeding the model’s original window size, significantly reducing computational and memory costs while maintaining performance. Extensive experiments on tasks like document understanding and few-shot learning with sequences up to 128K demonstrate that Activation Beacon achieves comparable accuracy to uncompressed models, with 2x inference acceleration and 8x KV cache reduction.
Key Results
- On LongBench long-text tasks, Activation Beacon maintains similar accuracy to full attention models while achieving 2x inference speedup and 8x KV cache savings. For example, in document understanding and multi-document summarization, performance drops are minimal even at 8x compression ratios, validating its effectiveness in preserving critical information.
- In Needle-in-a-Haystack retrieval tasks, the model accurately retrieves key information over sequences as long as 128K, outperforming baseline compression methods like AutoCompressor and LongLMLingua, especially in multi-turn settings. This demonstrates its fine-grained compression and information retention capabilities.
- Training with randomly sampled compression ratios enables the model to adapt flexibly to various downstream requirements, balancing efficiency and accuracy across diverse tasks.
Significance
This work addresses the fundamental challenge of long-text processing in large language models by reducing quadratic self-attention complexity and KV cache memory. The progressive activation compression mechanism offers a scalable solution for tasks requiring understanding and reasoning over extremely long sequences, opening new horizons for applications in legal, scientific, and content creation domains. Its ability to preserve information fidelity while significantly boosting efficiency marks a major step forward in long-context NLP, facilitating deployment of large models in resource-constrained environments and real-time scenarios.
Technical Contribution
The key technical innovation lies in the introduction of beacon tokens that encode and cache layer-wise activations, enabling progressive, fine-grained compression. Unlike soft prompts or global soft token methods, this approach supports multiple compression ratios and incremental updates, thanks to the caching mechanism. The training employs a compression-based auto-regressive loss, ensuring high-quality generation conditioned on compressed representations. The method also leverages reusing previous activations to avoid re-encoding, significantly reducing computational overhead, and supports multi-scale compression through random ratio sampling, enhancing versatility.
Novelty
This is the first work to implement progressive, layer-wise activation compression via interleaved beacon tokens in transformer models, enabling flexible, multi-ratio long sequence processing. Unlike prior soft prompt or sparse attention methods, Activation Beacon combines incremental distillation with activation caching, providing a scalable and information-preserving solution for ultra-long text handling. Its unique chunk-wise, progressive workflow and multi-ratio training set it apart from existing approaches.
Limitations
- The chunk-based segmentation may introduce boundary artifacts or information loss at extremely high compression ratios, especially for highly detailed or nuanced content. The method's robustness in dynamic or highly variable long sequences remains to be tested.
- Training complexity increases due to random ratio sampling, and performance may vary across different tasks or datasets. Further optimization is needed for real-time applications with dynamic input lengths.
- While reducing KV cache and FLOPs significantly, the additional overhead of encoding beacon tokens and managing cache may still pose challenges in latency-critical deployments. Future work should explore hardware acceleration and adaptive compression strategies.
Future Work
Future directions include developing adaptive, context-aware compression ratios based on input complexity, integrating sparse attention mechanisms for further efficiency, and extending the approach to multi-modal long sequences such as video or multi-source data. Additionally, exploring hardware-aware implementations and real-time adaptive schemes could further enhance practical deployment.
AI Executive Summary
Processing long sequences has become a critical challenge in deploying large language models (LLMs) for real-world applications. Traditional transformer architectures, while powerful, suffer from quadratic complexity in self-attention, limiting their efficiency when handling sequences beyond a few thousand tokens. Existing solutions such as sparse attention and soft prompts have made progress but face limitations in information preservation, flexibility, and scalability. To address these issues, this paper introduces Activation Beacon, a novel mechanism that progressively compresses activations through interleaved beacon tokens, enabling models to process sequences up to 128K tokens efficiently.
Activation Beacon works by dividing long inputs into manageable chunks, inserting beacon tokens that distill and cache layer-wise activations, and reusing these cached activations for subsequent chunks. This progressive, fine-grained compression allows the model to maintain high fidelity of critical information while significantly reducing computational and memory costs. The training employs a compression-based auto-regressive loss, with randomly sampled ratios to support flexible downstream configurations. Extensive experiments on long-document understanding, multi-document summarization, and retrieval tasks demonstrate that Activation Beacon achieves performance comparable to uncompressed models, with a 2x inference speedup and 8x KV cache reduction.
This work fundamentally advances the state-of-the-art in long-text processing, offering a scalable, efficient, and adaptable solution that can be integrated into existing transformer architectures. Its ability to handle sequences far exceeding the original window size opens new possibilities for AI applications in legal, scientific, and content creation fields. Despite some limitations in boundary artifacts and training complexity, the proposed method sets a new benchmark for long sequence modeling, with promising directions for future research in adaptive compression, multi-modal extension, and hardware acceleration.
Deep Analysis
Background
近年来，随着Transformer架构的普及，长文本处理成为研究重点。早期方法如旋转位置编码（RoPE）和稀疏注意力（Sparse Attention）在缓解计算瓶颈方面取得一定成效，但仍难以满足超长文本的高效处理需求。软提示（Soft Prompt）和KV压缩（KV Compression）等技术通过简化表示或压缩激活信息，降低了成本，但在信息完整性和灵活性方面存在不足。随着模型规模不断扩大，处理超长文本（如128K）成为新的挑战，亟需创新机制突破现有瓶颈。
Core Problem
长文本压缩的核心难题在于如何在极大缩减信息的同时，保持关键信息的完整性。现有软提示和全局压缩方法因缺乏细粒度控制，难以应对复杂任务，尤其在超长序列中表现不佳。软提示的重编码和硬性剪枝带来效率瓶颈，限制了实际应用。如何设计一种既能高效压缩，又能支持多比例、多场景的机制，成为亟待解决的问题。
Innovation
本研究的创新点在于引入逐步激活压缩机制，通过beacon tokens实现细粒度、多比例的长文本压缩。具体包括：1）在每层引入beacon tokens，逐步压缩激活信息；2）采用chunk划分策略，将长文本分块编码；3）训练中随机采样压缩比例，增强模型适应性；4）利用激活缓存，避免重复编码，提升效率。这些创新突破了软提示和全局压缩的局限，为超长文本处理提供了新思路。
Methodology
- �� 将长文本划分为等长块，插入beacon tokens，形成压缩输入；
- �� 每块内，利用自注意力机制，将原始激活逐步压缩到beacon tokens中；
- �� 逐块编码，激活缓存逐步积累，支持超长文本；
- �� 训练中，随机采样不同压缩比例，优化多样性；
- �� 在推理中，利用缓存激活，避免重复编码；
- �� 采用压缩自回归训练，确保生成质量；
- �� 激活重用机制支持多尺度压缩。
Experiments
采用LongBench和Needle-in-a-Haystack等长文本任务，验证模型在128K超长文本中的性能。对比全注意力和其他压缩方法，指标包括准确率、推理速度和KV缓存节省。模型在最高8倍压缩比下，保持与未压缩模型相当的性能，推理速度提升2倍，KV缓存节省8倍。多轮问答和信息检索中表现优异，验证了细粒度压缩的有效性。
Results
在长文本任务中，Activation Beacon在保持性能的同时实现了2倍推理加速和8倍KV缓存节省，优于现有方案。具体表现为在文档理解和少样本学习中，压缩比达8倍时仍能保持较高准确率。多轮问答中，模型能准确检索关键信息，验证了其信息保留能力。随机采样不同压缩比例训练，增强了模型的适应性和灵活性。
Applications
该技术适用于超长文档理解、内容生成、知识存储等场景，尤其在需要处理超出模型原有窗口的文本时表现出色。可广泛应用于法律、科研、内容审核等行业，提升大模型的实际效能。未来，结合稀疏注意力和量化技术，有望进一步降低成本，推动行业普及。
Limitations & Outlook
当前方法依赖chunk划分，可能在极端超长文本或动态场景中表现不稳定。压缩比例过高时，信息可能丢失，影响生成质量。训练复杂度增加，模型对不同任务的适应性仍需验证。未来需优化压缩策略，提升鲁棒性和泛化能力。
Plain Language Accessible to non-experts
想象你在整理一大堆杂乱的文件，要把它们变得更容易找到重要内容。传统方法就像把所有文件扔进一个箱子，等你需要找东西时再翻半天。而Activation Beacon就像在每堆文件中放置标签（beacon tokens），每个标签都能记住那堆文件的重点信息。每次整理时，你只需看标签，就能快速了解内容大意，而不用翻遍所有文件。这样，不仅节省时间，还能处理比平时多得多的文件。这个方法逐步压缩信息，把复杂内容变得简单明了，就像用标签帮你整理出一份精炼的摘要，方便快速检索和理解。
Abstract
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose Activation Beacon, a plug-in module for transformer-based LLMs that targets effective, efficient, and flexible compression of long contexts. To achieve this, our method introduces the following technical designs. 1) We directly compress the activations (i.e. keys and values at every layer), rather than leveraging soft prompts to relay information (which constitute a major bottleneck to encapsulate the complex information within long contexts). 2) We tailor the compression workflow, where each fine-grained input unit is progressively compressed, enabling high-quality compression and efficient computation during both training and inference. 3) We train the model through compression-based auto-regression, making full use of plain texts and instructional data to optimize the model's compression performance. 4) During training, we randomly sample a compression ratio at each step, teaching the model to support a wide range of compression configurations. Extensive evaluations are conducted on various long-context tasks whose lengths (e.g., 128K) may far exceed the maximum training length (20K), such as document understanding, few-shot learning, and Needle-in-a-Haystack. Whilst existing methods struggle to handle these challenging tasks, Activation Beacon maintains a comparable performance to the uncompressed baseline across various scenarios, achieving a 2x acceleration in inference time and an 8x reduction of memory costs for KV cache. Our data, model, and code have been released at \url{https://github.com/FlagOpen/FlagEmbedding/}.