EarlyTom: Early Token Compression Completes Fast Video Understanding
EarlyTom performs early-stage token compression inside the vision encoder, reducing TTFT by 2.65× and FLOPs by 61%, maintaining accuracy.
Key Findings
Methodology
EarlyTom introduces a training-free token compression framework with two main components: •In-encoder frame merging, which dynamically fuses redundant frames based on streaming similarity to reduce early visual tokens. •Decoupled spatial token selection, dividing features into dynamic and static parts, applying global Top-K sampling for dynamic frames and local window sampling for static frames. •System co-design offloading static token selection to CPU, leveraging heterogeneous hardware. This approach optimizes the visual encoding process, achieving up to 2.65× TTFT reduction and 61% FLOPs savings on LLaVA-OneVision-7B, while preserving accuracy.
Key Results
- On a single NVIDIA A100 GPU, EarlyTom reduces TTFT from 889ms to 336ms, a 2.65× speedup, with FLOPs decreasing by 61%, and maintains comparable accuracy to full-token baselines.
- Across benchmarks like MVBench, EgoSchema, LongVideoBench, and VideoMME, it delivers faster inference and higher throughput without accuracy loss, demonstrating strong practical benefits.
Significance
This work addresses the critical bottleneck in video large models—the high latency during visual encoding. By enabling early token compression within the encoder, it significantly improves inference speed and computational efficiency. This advancement makes deploying large-scale video understanding models in real-world scenarios feasible, bridging the gap between research and practical applications, and paving the way for real-time video AI systems.
Technical Contribution
The core innovations include: •A novel, training-free frame merging mechanism inside the vision encoder that reduces redundant tokens early. •A decoupled spatial token selection strategy that combines global importance sampling for dynamic frames and local window sampling for static frames, avoiding bias and preserving spatial distribution. •A system-level heterogeneous computation scheme that shifts static token filtering to CPU, optimizing hardware utilization. These contributions collectively enable high-speed, accurate video inference with minimal overhead.
Novelty
This is the first work to implement early token compression directly within the vision encoder using a dynamic, streaming-based frame merging approach. Unlike prior methods that compress tokens post-encoding or within the LLM, EarlyTom optimizes the entire encoding pipeline, achieving unprecedented speedups while maintaining accuracy. Its decoupled spatial sampling further distinguishes it from existing holistic compression techniques, offering a new paradigm for efficient video large models.
Limitations
- The frame merging relies on similarity metrics, which may be less effective in highly dynamic scenes with rapid content changes. This could lead to suboptimal compression or information loss.
- Static window sampling might introduce bias in complex scenes with diverse spatial features, affecting understanding accuracy.
- The heterogeneous CPU-GPU system design, while effective, may face challenges in cross-platform deployment and scalability, requiring further optimization.
Future Work
Future directions include integrating self-supervised learning to adaptively refine frame merging and sampling strategies, enhancing robustness in dynamic scenes. Extending the framework to multi-modal inputs and more complex tasks, such as video captioning or action recognition, is also promising. Additionally, optimizing hardware deployment for edge devices will broaden practical applications.
AI Executive Summary
Large-scale video understanding models, powered by transformer architectures, have achieved remarkable performance but face significant efficiency challenges. The core bottleneck lies in the visual encoding stage, where processing massive visual tokens incurs high latency and computational cost. Existing compression approaches mainly operate after encoding or within the language model, leaving the encoding phase largely unoptimized. This limits the deployment of such models in real-time applications like surveillance, autonomous driving, and content retrieval.
In this context, the authors propose EarlyTom, a novel, training-free framework that performs early-stage token compression directly inside the vision encoder. The key innovation is a streaming-based frame merging strategy that dynamically fuses similar frames, reducing temporal redundancy early in the pipeline. Complementing this, a decoupled spatial token selection mechanism divides features into dynamic and static parts, applying global importance sampling and local window sampling respectively, to maximize compression while preserving semantic fidelity.
Crucially, the system employs a heterogeneous computation scheme, offloading static token filtering to CPU, which alleviates GPU load and accelerates inference. Extensive experiments on benchmarks such as MVBench, EgoSchema, LongVideoBench, and VideoMME demonstrate that EarlyTom reduces time-to-first-token by 2.65× and FLOPs by 61%, with minimal accuracy loss. On the LLaVA-OneVision-7B model, this translates into faster, more practical deployment, enabling near real-time video understanding.
This work advances the state-of-the-art in efficient video large models, providing a scalable solution to the latency bottleneck. Its innovative combination of early visual token merging, decoupled spatial sampling, and system-level optimization sets a new benchmark for practical, high-performance video AI. Future research may extend these ideas to multi-modal tasks, adaptive learning, and edge deployment, further broadening the impact of this approach.
Deep Analysis
Background
近年来，视频理解技术迅速发展，Transformer架构成为主流。代表性工作如Video Swin Transformer、TimeSformer在性能提升方面取得突破，但计算成本依然高昂。随着大模型的兴起，如何在保证精度的同时降低推理延迟成为关键难题。现有压缩方法多在模型后端进行，忽视了视觉编码阶段的潜在冗余，导致整体效率提升有限。近年来，研究者开始关注在编码器内部实现压缩，但缺乏高效、无需训练的方案，难以满足实际部署需求。
Core Problem
核心问题在于视觉编码阶段耗时过长，尤其是在大规模视频场景中，TTFT成为瓶颈。传统压缩多在模型后端进行，未充分利用编码器内部的冗余信息，导致整体效率难以突破。如何在不影响模型精度的前提下，提前压缩冗余令牌，成为提升推理速度的关键。现有技术难以兼顾压缩效果和实时性，亟需创新方案解决这一瓶颈。
Innovation
本研究的创新点包括：1）提出无需训练的内层帧合并机制，利用流式相似性动态融合冗余帧，显著减少早期令牌数；2）引入解耦空间令牌选择策略，将动态与静态帧分别采用全局重要性采样和局部窗口采样，避免偏差，提升压缩效果；3）系统协同设计，将静态令牌筛选迁移到CPU端，充分利用异构硬件资源。这些创新实现了在保持模型精度的基础上，大幅提升推理速度和效率。
Methodology
- ��在视觉编码器中引入流式帧分割，根据余弦相似度动态划分视频段。•在中间帧采用局部最优合并策略，融合相似帧以减少冗余。•利用加权融合增强合并帧的语义表达。•将融合后的视频特征划分为动态和静态两类，分别采用全局Top-K和局部窗口Top-K进行令牌压缩。•将静态令牌筛选部分迁移到CPU端，利用异构计算资源提升效率。•最终，将压缩后的令牌重新组合，输入到LLM进行推理。整个流程实现了早期令牌压缩、减少延迟和提升吞吐。
Experiments
在MVBench、EgoSchema、LongVideoBench和VideoMME四个公开数据集上进行评估。采用LLaVA-OneVision-7B模型，比较不同压缩比例下的TTFT、FLOPs和准确率。超参数包括：流式分割阈值、合并阈值、Top-K比例等。通过消融实验验证帧合并策略和空间采样的贡献。结果显示，10%令牌保留率下，TTFT降低2.65倍，FLOPs减少61%，准确率仅下降1%。多场景测试证明了方法的鲁棒性和优越性。
Results
实验结果显示，EarlyTom在保持模型性能的同时，显著提升推理速度和计算效率。具体而言，在LLaVA-OneVision-7B模型上，TTFT由889ms降至336ms，提升2.65倍，FLOPs减少61%。在多个视频理解任务中，准确率与全令牌基线相差不大，验证了压缩策略的有效性。与现有方法如HoliTom、VisionZip相比，EarlyTom在速度和效率方面均优越，展现出极强的实用潜力。
Applications
该技术适用于需要实时视频分析的场景，如智能监控、自动驾驶、视频检索等。通过显著降低推理延迟，满足工业级应用的实时性要求。未来可结合边缘计算设备，推动端侧高效视频理解系统的落地。长远来看，EarlyTom为大规模视频数据的智能处理提供了基础，有望在视频内容生成、虚拟现实等领域发挥重要作用。
Limitations & Outlook
目前方法依赖视频帧的相似性进行流式分割，在快速变化或高动态场景中可能表现不足。此外，静态帧的局部窗口采样可能引入信息偏差，影响理解效果。系统在不同硬件平台上的适应性和优化空间仍需探索。未来需结合自监督学习和多模态信息，进一步提升鲁棒性和泛化能力。
Plain Language Accessible to non-experts
想象你在整理一堆照片，想要快速找到最重要的几张。传统方法会逐一检查每张照片，耗时很长。EarlyTom就像一个聪明的助手，它会先把相似的照片合成一组，减少重复内容，然后只挑出最有代表性的几张。这么做可以大大缩短整理时间，还能保证你看到的内容和原来一样丰富。它在视频理解中也是一样的，提前把冗余信息压缩掉，让电脑更快理解视频内容，就像你用更少的时间看完一堆照片却依然记住了重点。
ELI14 Explained like you're 14
想象你在看一大堆视频片段，平时要一帧一帧地分析，特别慢。EarlyTom就像个聪明的朋友，它会在你看视频的时候，把那些重复的画面合成一块，只留下最重要的部分。这样，你就不用花那么多时间，也能快点理解视频内容。它还会根据每个画面的重要性，挑出最关键的部分，避免被无关紧要的内容干扰。结果就是，电脑能更快地看懂视频，还能用更少的计算资源，像你用更少的时间看完一部电影，但依然记住了故事的重点。这让视频处理变得更快、更高效，也更实用。
Abstract
Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.