Patch Slimming for Efficient Vision Transformers
Patch Slimming removes redundant ViT patches, cutting DeiT-Ti FLOPs by 53.8% with only a 0.1-point ImageNet Top-1 drop.
Key Findings
Methodology
Patch Slimming performs top-down pruning from the final transformer layer to the first. It initially retains the class token, estimates each earlier patch’s effect on the final effective output through accumulated downstream attention, ranks patches by an impact score, and progressively restores the most important ones. Each selection step fine-tunes the current block and stops when reconstruction error meets tolerance ε. DPS-ViT adds a lightweight module G that predicts input-dependent patch scores.
Key Results
- On ImageNet, DeiT-Ti improves efficiency from a 1.3G-FLOP, 72.2% Top-1 baseline to PS-ViT at 0.7G and 72.0%; DPS-ViT reaches 0.6G, a 53.8% FLOP reduction, 72.1% Top-1, and 3,639 images/s versus 2,536.
- For DeiT-S, DPS-ViT reduces 4.6G to 2.4G FLOPs, a 47.8% reduction, while Top-1 falls only from 79.8% to 79.5%. For DeiT-B, it reduces 17.6G to 9.4G and retains 81.6% Top-1.
- At the same 2.6G budget on DeiT-S, learned PS-ViT achieves 79.4% Top-1, whereas uniform pruning reaches only 77.2%, showing that task- and layer-aware selection is superior to fixed-rate deletion.
Significance
The paper shifts transformer compression from channels, heads, and embedding dimensions toward spatial patch redundancy. This directly addresses the computational burden that limits ViT deployment on mobile, IoT, and real-time systems. Its broader contribution is conceptual: redundancy is not merely a property of weights or dimensions, but of information flow across tokens and layers. By connecting final task relevance with backward propagation constraints, the method obtains substantial structured savings without requiring specialized hardware, while retaining almost all classification accuracy.
Technical Contribution
The central metric is s_{t,i}=Σ_h||A_t^h[:,i]U_t^h[i,:]||²_F, where A is cumulative downstream attention propagation and U=P_t^h|Z_{t−1}|. The algorithm searches backward from the output, forces shallow layers to preserve deeper-layer positions, and controls sparsity using a reconstruction-error tolerance. This explicitly handles both one-to-one shortcut correspondence and global MSA aggregation. It therefore differs fundamentally from directly transferring CNN channel pruning, BERT word-vector removal, or embedding-dimension pruning.
Novelty
The main novelty is to define and exploit cross-layer patch-information redundancy in vision transformers rather than pruning only channels, heads, or dimensions. The inverse, top-down procedure uses the final class-token objective to guide earlier layers. This is especially important because visual patches have spatial correspondence and self-attention creates information paths that conventional independent pruning can disrupt.
Limitations
- The score is an approximation based on training-set statistics and attention propagation. Static masks may delete useful regions for out-of-distribution images, fine-grained recognition, heavy occlusion, or tasks requiring many local outputs.
- The offline search requires repeated score computation, block fine-tuning, and error evaluation. DPS-ViT adds a runtime predictor, and practical latency gains depend on hardware and compiler support for variable-length token sequences.
Future Work
Future work should develop task-specific constraints for detection, segmentation, pose estimation, and video, and jointly prune patches, heads, MLP dimensions, and numerical precision. Hardware-aware token layouts could better translate FLOP savings into latency. Comparisons with gradient, perturbation, and distillation criteria may improve robustness, while end-to-end training of dynamic routers could reduce the current search and calibration overhead.
AI Executive Summary
Vision Transformers achieve strong visual recognition by processing an image as a sequence of patches, but they normally carry every patch through every self-attention and MLP block. This is expensive, especially because attention scales quadratically with token count. CNN channel pruning, BERT word-vector removal, and pooling-based designs such as HVT do not fully address the spatial correspondence and information-flow structure of visual tokens.
Patch Slimming introduces a backward, layer-wise strategy. It starts at the final layer, keeps the class token required for classification, and estimates how strongly each earlier patch affects the final effective representation through downstream attention. Patches are ranked by a significance score, selected incrementally, and calibrated with block fine-tuning until reconstruction error falls below tolerance ε. Static PS-ViT uses fixed masks; dynamic DPS-ViT uses a lightweight module G to select patches per image.
On ImageNet, PS-ViT reduces DeiT-Ti from 1.3G to 0.7G FLOPs while retaining 72.0% versus 72.2% Top-1 accuracy. DPS-ViT reaches 0.6G FLOPs, a 53.8% reduction, 72.1% Top-1, and 3,639 images/s. On DeiT-S it retains 79.5% accuracy at 2.4G FLOPs. The work shows that ViT efficiency can be improved by removing redundant information carriers, not merely shrinking dimensions. Open issues include search cost, hardware support for variable token counts, and transfer beyond image classification.
Deep Analysis
Background
ViT, DeiT, T2T-ViT, and LV-ViT established strong transformer performance in vision, but their MSA and MLP blocks remain computationally expensive. Prior compression includes SCOP-style CNN channel pruning, PoWER for BERT word-vector elimination, VTP for embedding dimensions, and HVT for progressive pooling. These methods mainly compress channels, heads, dimensions, or resolution; they do not directly exploit the high similarity that emerges among visual patches as attention repeatedly aggregates them across layers.
Core Problem
For N patches and embedding dimension d, an MSA block costs 2N²d+4Nd², while a two-layer MLP costs 2Ndd′. Patch similarity rises with depth and exceeds 0.8 on deep ViT-Base layers, indicating repeated computation. Yet visual tokens have one-to-one spatial correspondence across layers, and shortcut connections preserve this alignment. Independent deletion can therefore break information paths, creating the need for a task-aware criterion that reduces tokens while preserving cross-layer propagation.
Innovation
- �� Top-down pruning: process layers from output to input and force shallow layers to retain positions preserved deeper in the network.
- �� Impact estimation: compute s_{t,i}=Σ_h||A_t^h[:,i]U_t^h[i,:]||²_F, using cumulative attention A and current value propagation U.
- �� Error-controlled search: add patches in score order, fine-tune the current block, and stop when the next-layer reconstruction error is below ε.
- �� Dynamic extension: DPS-ViT uses a small module G to predict per-instance significance and select different patches for different images.
Methodology
- �� Input: a pretrained ViT, a training subset, tolerance ε, and search granularity r′.
- �� Initialization: at layer L, set m_{L,1}=1 for the class token and set other entries to zero.
- �� Attention analysis: compute P_t^h=softmax(QK^T/√d), A_t^h=∏_{l=t+1}^Ldiag(m_l)P_l^h, and U_t^h=P_t^h|Z_{t−1}|; average scores over sampled training images.
- �� Selection: initialize m_l from m_{l+1}, then activate the highest-scoring patches in increments of r′.
- �� Calibration: fine-tune the current block after each increment and measure E_{l+1}=||diag(m_{l+1})(Ẑ−Z)||²_F; stop when E≤ε.
- �� Deployment: compute queries, attention, and MLP operations only for effective patches, align outputs through shortcuts or zero padding, then fine-tune the complete pruned model. DPS-ViT predicts scores using G at inference.
Experiments
The evaluation uses ImageNet ILSVRC2012, with 1.2M training images, 5,000 validation images, and 1,000 classes. Models include DeiT-Ti/S/B, T2T-ViT-14, and LV-ViT. Baselines are SCOP, PoWER, HVT, and VTP. Tolerance ε is 0.01 or 0.02, search granularity is 10, and each selection step uses three epochs of block fine-tuning. Experiments run with PyTorch and MindSpore on NVIDIA V100 GPUs and report Top-1, Top-5, FLOPs, and throughput. Uniform pruning is also tested as an ablation.
Results
On DeiT-Ti, PS-ViT obtains 72.0% Top-1 at 0.7G FLOPs, compared with 68.9% for SCOP and 69.4% for PoWER at similar cost; DPS-ViT reaches 72.1% at 0.6G. On DeiT-B, DPS-ViT delivers 81.6% Top-1 at 9.4G versus 81.8% at 17.6G. On T2T-ViT-14, DPS-ViT reaches 81.3% at 3.1G versus 81.5% at 5.2G. At 2.6G on DeiT-S, uniform pruning gives 77.2%, while PS-ViT gives 79.4%.
Applications
The method is suited to image-classification servers, mobile cameras, edge vision, and IoT devices, provided that inference stacks support token indexing, sparse execution, or variable-length sequences. Static PS-ViT is easier to deploy on fixed hardware; DPS-ViT is preferable when image complexity varies substantially. Direct use in detection and segmentation requires redesigned task losses because many spatial outputs must remain available.
Limitations & Outlook
The evidence is concentrated on ImageNet classification, where a single class token makes output reconstruction relatively simple. Dense prediction, video, and fine-grained tasks may need several local regions simultaneously. Attention-product scoring and layer-wise search impose offline costs, while the dynamic predictor adds runtime overhead. The theoretical score is an approximation under Lipschitz assumptions rather than a global optimum. Finally, lower FLOPs may not yield proportional latency reductions without compiler and hardware support for irregular token counts.
Plain Language Accessible to non-experts
Imagine a large warehouse whose shelves are image patches. A team of workers—the transformer—walks through every shelf in repeated rounds, allowing shelves to exchange information before preparing a final shipping label. The trouble is that many shelves contain nearly identical goods, yet workers inspect all of them every time.
Patch Slimming first looks at the final shipping label and asks which shelves actually influenced it. It then walks backward through the warehouse. If a shelf’s information still travels through later rounds and affects the label, it stays. If its information is repeated or has no meaningful downstream effect, it can be removed. Shelves kept in later rounds must also remain in earlier rounds, so the communication route is not cut.
After removing a batch, the system briefly retrains the remaining workers so they can coordinate again. PS-ViT uses the same shelf plan for every shipment; DPS-ViT acts like a supervisor who chooses a different inspection list for each image. On ImageNet, this allows DeiT-Ti to perform up to 53.8% fewer calculations while losing only 0.1 percentage point of Top-1 accuracy.
ELI14 Explained like you're 14
Imagine a picture-guessing game where the screen is divided into many squares. Each square is like a student who says what they see, and after several rounds the team leader gives the teacher a final answer. The problem is that some students are looking at the same boring wall or sky. Making everyone talk every round wastes time!
Patch Slimming works backward from the answer. It asks which students really helped the leader. Helpful students keep talking; students whose information is repeated can skip some rounds. But if a student’s position is still useful later, that position must stay earlier too, otherwise the message route gets broken.
The algorithm does not simply delete half the squares. It gives each square an importance score, removes some, and checks whether the important final information changed too much. If it did, more useful squares are added back. DPS-ViT is even more flexible: it chooses a different group for every picture because a beach photo and a crowded classroom need different clues.
The result is impressive: on ImageNet, DeiT-Ti can reduce computation by 53.8% with DPS-ViT, while Top-1 accuracy remains 72.1% instead of the original 72.2%. In other words, the model skips lots of repeated work but still makes almost the same guesses!
Glossary
Vision Transformer (ViT)
A neural network that splits an image into patches and models their relationships with self-attention. It offers flexible global interaction but can be computationally expensive.
The paper compresses DeiT, T2T-ViT, and LV-ViT models.
Patch
A local image region represented as a token embedding. Its token retains a corresponding spatial position through transformer layers.
Patch Slimming identifies and removes redundant patch tokens.
Multi-head self-attention (MSA)
A module that computes relationships using queries, keys, and values, then aggregates information across tokens. Its attention matrix connects every patch with others.
Downstream MSA propagation is used to estimate patch impact.
Top-down pruning
A pruning strategy that determines useful units in later layers first and propagates constraints toward earlier layers. It preserves cross-layer dependencies.
The algorithm processes layer L through layer 1 in reverse order.
Significance score
A numerical estimate of how much removing a patch would affect the final effective output. Higher scores indicate greater importance.
The paper defines s_{t,i}=Σ_h||A_t^h[:,i]U_t^h[i,:]||²_F.
FLOPs
The number of floating-point operations used as an approximate measure of computational cost. FLOPs do not always equal latency because memory and hardware effects also matter.
The experiments report FLOP reductions and throughput changes.
Open Questions Unanswered questions from this research
- 1 The evidence is mainly from ImageNet classification. Detection, segmentation, video, and fine-grained recognition may require many local regions, so class-token reconstruction may not be sufficient.
- 2 The score depends on attention and training-set statistics. Robust selection under distribution shift, occlusion, unusual backgrounds, or adversarially difficult images remains insufficiently tested.
- 3 Whether FLOP savings become end-to-end latency gains depends on compiler and hardware support for variable-length tokens and irregular matrix operations.
Applications
Immediate Applications
Mobile image classification
Convert a pretrained DeiT into PS-ViT and deploy its static mask to reduce MSA and MLP computation. Suitable for camera recognition and photo search; the runtime must support token indexing, followed by calibration fine-tuning.
Adaptive edge vision
Use DPS-ViT to select fewer patches for simple images and more for complex scenes. This suits cameras and IoT nodes, but requires the lightweight predictor G and an efficient implementation for variable token counts.
Long-term Vision
Multi-task visual token routing
Jointly optimize patch, head, MLP, and quantization choices to build task-aware routers for detection, segmentation, and video. The system could allocate computation dynamically under an explicit accuracy budget.
Abstract
This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a top-down paradigm. We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers. For each layer, the impact of a patch on the final output feature is approximated and patches with less impact will be removed. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers without affecting their performances. For example, over 45% FLOPs of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the ImageNet dataset.