Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption
Proposes ISPA, compressing KV cache by 50% via parametric absorption, with near-lossless visual quality.
Key Findings
Methodology
This paper introduces ISPA, transforming long-range memory compression into parameter distillation. During a brief warmup, the model monitors the discrepancy between global and local attention outputs. Using a closed-form least squares solution, it computes instance-specific weight modulation to internalize long-term context into model weights. The approach involves switching some layers from full attention to local attention, then absorbing the missing history into weights W+ΔW, avoiding cache growth. A decomposable attention mechanism leverages hardware-supported Log-Sum-Exp states to fuse attention signals efficiently during inference, enabling real-time operation. Experiments across models from 1.3B to 14B parameters show up to 50% cache removal with negligible quality loss, significantly reducing memory and increasing speed.
Key Results
- On 30-second videos, ISPA achieves near lossless compression, evicting up to 50% of KV cache with less than 1% visual quality deviation. Peak memory reduces by 23.7GB in the 14B Krea model, with a 1.86× inference speedup when combined with quantization.
- Across metrics like aesthetics, background consistency, and motion smoothness, performance remains stable or slightly improves at 50% cache removal, demonstrating effective long-range dependency preservation.
- Dynamic layer selection based on warmup reconstruction error allows the model to adapt to different scene dynamics, ensuring robustness and flexibility.
Significance
This work addresses the fundamental memory bottleneck in long video autoregressive models by internalizing historical context into model weights, enabling scalable, real-time long video synthesis. It shifts the paradigm from external cache management to internal parameter-based memory, opening new avenues for efficient, high-fidelity generation on resource-constrained hardware, and influencing future research in model internal memory management.
Technical Contribution
The core innovation is the parametric absorption mechanism, which replaces cache pruning with instance-specific weight modulation. It employs a closed-form least squares solution during inference, avoiding gradient-based optimization, and integrates a decomposable attention module for efficient global information fusion. This design allows models to internalize long-term dependencies dynamically, significantly reducing memory footprint while maintaining quality and speed.
Novelty
This is the first work to formulate long-range memory compression as a parametric absorption problem, leveraging instance-specific linear weight modulation. Unlike prior methods relying solely on cache pruning or external memory, it internalizes long-term context directly into model weights, supported by a hardware-friendly attention decomposition, representing a paradigm shift in streaming generative modeling.
Limitations
- The method depends on warmup monitoring, which may be less effective during abrupt scene changes or highly dynamic sequences, requiring re-calibration.
- Parameter absorption may not fully replace full attention in scenarios with extremely long-range dependencies or highly complex scenes, leading to potential quality trade-offs.
- Model tuning for different architectures and tasks remains necessary, and generalization across diverse domains needs further validation.
Future Work
Future research could explore multi-stage, hierarchical parameter absorption strategies, adaptive layer selection mechanisms, and integration with hardware accelerators. Extending the approach to larger models and multi-modal tasks, as well as developing automatic scene-aware recalibration, will further enhance its robustness and applicability.
AI Executive Summary
Long video generation remains a challenging frontier in AI, constrained by the explosive growth of memory requirements for maintaining long-range dependencies. Traditional autoregressive models rely on external KV caches that grow linearly with sequence length, leading to memory overload and slower inference. This bottleneck hampers the deployment of high-fidelity, real-time long video synthesis in practical settings. Addressing this, the paper introduces the Instance-Specific Parametric Absorption (ISPA) framework, a novel approach that internalizes long-term historical information into model weights, effectively compressing memory without sacrificing quality.
The core idea involves a brief warmup phase during which the model monitors the discrepancy between full and local attention outputs across selected layers. Using a closed-form least squares solution, it computes instance-specific weight modulations (ΔW) that compensate for the missing historical context. Once computed, these modulations are applied to convert certain layers from full attention to local attention, and the historical KV caches for these layers are permanently evicted. This process effectively internalizes long-range dependencies into the model parameters, enabling subsequent inference to operate with significantly reduced memory.
To facilitate real-time operation, the authors design a decomposable attention mechanism that leverages hardware-supported Log-Sum-Exp states. This allows the model to fuse full and local attention signals efficiently during the forward pass, avoiding additional computational overhead. Extensive experiments across models from 1.3B to 14B parameters demonstrate that ISPA can remove up to 50% of KV cache with negligible (<1%) visual quality loss. In the largest models, this results in a peak memory reduction of 23.7GB and a speedup of 1.86× when combined with quantization.
This work represents a paradigm shift in memory management for streaming generative models, moving from external cache pruning to internal parametric consolidation. It opens new avenues for scalable, high-quality long video synthesis on resource-limited hardware, and suggests future directions including multi-layer absorption, adaptive layer selection, and hardware acceleration integration.
Deep Analysis
Background
Video generation技术经历了从早期基于GAN的模型到基于扩散和流模型的高质量合成。代表作如Video Diffusion Models、Flow-GAN等，解决了单帧质量和时间一致性问题，但在长视频生成中面临存储和计算瓶颈。自回归模型通过逐帧生成和KV缓存实现长序列，但随着序列增长，内存和速度成为限制。近年来，研究尝试通过剪枝、压缩等策略缓解，但依然难以兼顾长程依赖和效率。
Core Problem
核心问题在于KV缓存线性增长导致的内存瓶颈，限制了长视频的实时生成能力。传统方法通过丢弃冗余Token或eviction策略，破坏了长距离依赖，造成画面抖动和身份丧失。如何在保证长程依赖的同时，显著减少缓存存储，是当前难题。
Innovation
提出参数吸收（ISPA），在暖身阶段监测全局与局部注意差异，利用闭式最小二乘解动态调节模型线性投影参数，将长距离历史信息内在化为模型参数。避免缓存无限增长，结合可分解注意力技术，实时融合全局信息，确保模型在后续生成中无需大量KV缓存。
Methodology
- �� 在模型部分层采用全注意力与局部注意力双流机制，收集暖身阶段的全局与局部注意输出。
- �� 监测两者差异，形成残差信号，利用闭式最小二乘解计算调节系数∆W。
- �� 暖身结束后，将差异最大的K层转为局部注意力层，丢弃其KV缓存，利用调节后的W+∆W模拟长程历史。
- �� 采用可分解注意力机制，利用硬件支持的Log-Sum-Exp状态，实时融合全局与局部注意力，避免额外计算。
- �� 根据暖身阶段的重建误差动态选择层进行参数吸收，确保不同场景的适应性。
- �� 通过调节参数和层选择，实现模型的动态结构弹性，支持多次调节应对场景变化。
Experiments
在多模型（1.3B至14B参数）和任务（文本到视频、语音到视频）上验证，使用VBench-Long和MovieGen数据集，评估压缩比例、视觉质量和推理速度。不同压缩比例（如50%）对比，验证模型在保持质量的同时显著降低内存。进行消融实验，分析暖身层选择和调节效果。
Results
最高压缩50%的KV缓存几乎无视觉质量损失（误差<1%），在30秒视频中表现优异。压缩后峰值内存减少23.7GB，推理速度提升1.86倍。不同模型和指标（如美学、背景一致性）表现稳定，部分指标略有提升，显示参数吸收不仅节省空间，还改善长时间生成的稳定性。
Applications
适用于长视频实时生成、虚拟主播、动画制作等场景，尤其在硬件资源有限的边缘设备上，能实现高质量长序列生成。通过模型参数的内在化，减少对大规模存储的依赖，提升生成效率和稳定性。
Limitations & Outlook
目前方法依赖暖身阶段的监测，场景剧烈变化时可能需重新调节，存在适应性限制。参数调节在不同模型和任务中需调优，泛化能力尚待验证。参数吸收在极端长距离依赖场景中可能不足以完全替代全注意力，存在性能折中。
Plain Language Accessible to non-experts
想象你在厨房做饭，平时需要用很多调料和工具来做出不同的菜肴。每次做饭时，你会用到很多调料瓶（就像模型中的KV缓存）来记住之前用过的调料，但这些瓶子会占用很多空间，特别是你要做很多不同菜肴时。现在，有个聪明的厨师发明了一个办法，他把常用的调料提前装到一个特殊的瓶子里（相当于模型参数中的吸收机制），这样你就不用每次都拿出所有瓶子，只需要用这个特殊的瓶子就能做出味道一样的菜。这就像论文中的参数吸收技术，把长远的记忆“装”到模型里，既节省空间，又保证菜的味道（视频质量）不变。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你需要记住很多以前的动作和策略（就像模型的长距离记忆）。如果每次都把所有的动作都记在脑袋里，脑袋会变得很重，反应也变慢。于是，你的哥哥告诉你一个秘密：你可以把一些重要的策略提前写在一本笔记里（模型参数），这样你以后就不用一直翻旧笔记了，只要看一眼笔记本就知道该怎么做。这就像论文里的参数吸收，把长时间积累的记忆变成了模型的“笔记”，让它既快又记得牢。这样，你在玩游戏时既能保持连贯，又不用担心记忆不够用，真是太酷了！
Glossary
Key-Value Cache (KV缓存)
存储模型在生成过程中积累的历史信息，用于维护长序列的依赖关系。技术上是存储键值对的缓存机制。
在论文中，KV缓存随着生成时间线性增长，成为内存瓶颈。
Full-Attention (全注意力)
一种注意力机制，模型在每一层都对所有位置的Token进行全局交互，计算复杂度为O(F²)。
在模型中用于捕获长距离依赖，但计算成本高。
Local-Attention (局部注意力)
只关注邻近位置或有限范围内Token的注意力机制，降低计算复杂度。
在ISPA中用以替代全注意力，减少内存使用。
参数吸收 (Parametric Absorption)
将长距离历史信息通过线性调节参数内在化到模型权重中，避免存储大量KV Token。
论文创新点，将缓存压缩转为参数调节。
Decomposable Attention (可分解注意力)
利用硬件支持的Log-Sum-Exp状态，将全局注意力拆分为局部和历史两部分，实时融合。
实现无额外计算开销的全局信息采集。
Open Questions Unanswered questions from this research
- 1 如何在极端场景或剧烈变化的场景中，动态调节参数吸收的效果和稳定性仍需验证。
- 2 模型在更大规模（百亿参数以上）或多模态长视频中的表现和适应性尚未充分研究。
- 3 未来需探索多层次、多阶段的参数吸收策略，以应对更复杂的长程依赖需求。
Applications
Immediate Applications
长视频实时生成
在虚拟主播、动画制作中，利用参数吸收技术实现高质量长视频的快速生成，减少硬件资源依赖。
边缘设备视频处理
在硬件受限的设备上，通过模型参数内在化长程记忆，提升长序列生成的效率和稳定性。
Long-term Vision
普适长视频生成平台
结合参数吸收与硬件加速，打造面向大众的高效长视频生成平台，推动虚拟现实、增强现实等行业发展。
Abstract
Autoregressive (AR) streaming models have emerged as a powerful paradigm for long video generation. However, the linearly growing Key-Value (KV) cache poses a significant bottleneck, leading to memory overload and degraded inference throughput. A common compression method is to drop redundant KV tokens, which often breaks long-range dependencies, resulting in temporal flickering and identity loss. In this paper, we propose Instance-Specific Parametric Absorption (ISPA), a novel framework that shifts the KV cache compression from discarding to distilling. The core idea is to transit a subset of layers from Full-Attention (F-Layers) to memory-efficient Local-Attention (L-Layers) by "absorbing" historical context into the model's weights. Specifically, during a brief warmup phase, ISPA monitors the output discrepancy between global and local attention. At the transition point, we solve a closed-form least-squares problem to compute an instance-specific weight modulation that compensates for the missing history. Experiments across architectures (1.3B to 14B) demonstrate that ISPA can remove up to 50\% of the KV cache with near-lossless visual quality. We hope this perspective encourages future work to explore parametric memory consolidation beyond external token-level cache management for streaming generative models.