Swin Transformer V2: Scaling Up Capacity and Resolution
Swin Transformer V2 combines post-norm, cosine attention, and Log-CPB to scale vision models to 3B parameters and 84.0% ImageNet-V2 accuracy.
Key Findings
Methodology
The paper scales Swin Transformer along capacity, resolution, and data efficiency. Residual post-normalization limits activation accumulation; scaled cosine attention replaces dot-product attention with cosine similarity divided by a learnable temperature τ; Log-CPB uses a two-layer MLP to generate relative position biases from log-spaced coordinates. SimMIM reduces label dependence, while ZeRO, activation checkpointing, and sequential attention enable 658M-parameter H and 3B-parameter G models.
Key Results
- SwinV2-G reaches 84.0% top-1 on ImageNet-V2, exceeding the previous 83.3% best result by 0.7 points. It obtains 63.1 box AP and 54.4 mask AP on COCO test-dev, 59.9 mIoU on ADE20K, and 86.8% top-1 on Kinetics-400.
- The model uses about 70M labeled images, roughly 1/40 of the JFT-3B practice, while supporting 1536×1536 detection inputs and 320×320×8 video inputs. This demonstrates joint scaling of model capacity and task resolution.
- In Swin-T ablations, Log-CPB transferred from 8×8 to larger windows more effectively: at a 24×24 window, ImageNet fine-tuning reached 84.2% versus 83.2% for parameterized bias; ADE20K at window 32 reached 47.8 mIoU, a 3.3-point gain.
Significance
The work moves vision Transformers from hundreds of millions of parameters and classification-focused evaluation toward a 3B-parameter, multi-task backbone. It addresses three persistent bottlenecks together: unstable optimization, poor transfer across pre-training and fine-tuning resolutions, and dependence on enormous labeled datasets. The results suggest that hierarchical windowed architectures combined with self-supervision can provide a practical route toward large general-purpose visual models and future vision-language systems.
Technical Contribution
Residual post-normalization normalizes each residual branch before merging it into the main stream, reducing layer-wise amplitude growth. Scaled cosine attention computes cos(q,k)/τ+B rather than qᵀk/√d, making logits less sensitive to input magnitude. Log-CPB generates B(Δx,Δy) through an MLP using sign(x)log(1+|x|) coordinates, reducing extrapolation. ZeRO stage-1, checkpointing, and sequential attention complete the algorithm–systems design on A100-40G GPUs.
Novelty
The novelty is not merely a larger backbone, but an integrated scaling recipe. Compared with Swin V1's pre-normalization, dot-product attention, and interpolated parameter tables, Swin V2 provides more stable optimization and a continuous position function that naturally supports changing windows. SimMIM further shifts large-scale vision pre-training away from exclusive reliance on massive classification-labeled datasets.
Limitations
- SwinV2-G relies on a private 70M-image ImageNet-22K-ext collection, substantial compute, and specialized data deduplication, making exact reproduction difficult for ordinary laboratories.
- The evaluation emphasizes supervised fine-tuning; zero-shot, few-shot, robustness, inference latency, energy use, and deployment cost are not systematically characterized.
- Comparisons differ in data, resolution, training length, and compute, so absolute gains should not always be interpreted as controlled architectural improvements.
Future Work
Future work should test open and smaller datasets, parameter-efficient transfer, and multimodal pre-training. The community should evaluate Log-CPB on irregular grids, 3D vision, and long videos, while establishing scaling laws linking parameters, data, resolution, compute, energy, and downstream accuracy.
AI Executive Summary
Large language models improve as they scale, but vision models face a harsher combination of instability, resolution mismatch, and label hunger. Swin Transformer V2 asks whether a hierarchical visual backbone can be enlarged in the same spirit. The challenge is substantial: in large pre-normalized Transformers, residual activations accumulate across depth; low-resolution pre-training does not transfer cleanly to high-resolution detection or segmentation; and billion-scale models appear to demand enormous labeled datasets.
The paper introduces three architectural changes. Residual post-normalization places LayerNorm after each residual branch, stabilizing signal amplitudes. Scaled cosine attention replaces dot products with cosine similarity divided by a learnable temperature, preventing a few pairs from dominating attention. Log-CPB uses a small MLP over log-spaced relative coordinates, producing position biases for arbitrary windows rather than interpolating a fixed table. SimMIM supplies self-supervised pre-training, while ZeRO, activation checkpointing, and sequential attention control memory use.
The resulting SwinV2-G has 3B parameters and operates at up to 1536×1536 resolution. It achieves 84.0% on ImageNet-V2, 63.1/54.4 box/mask AP on COCO, 59.9 mIoU on ADE20K, and 86.8% on Kinetics-400. The authors report using about 40 times less labeled data than JFT-3B-based practice. The work is therefore an important scaling demonstration, although its private data, high compute requirements, limited zero-shot analysis, and incomplete efficiency reporting constrain reproducibility and broader conclusions.
Deep Analysis
Background
Swin Transformer established hierarchical features, local windows, and shifted-window attention as an effective backbone for ImageNet, COCO, and ADE20K. Meanwhile, ViT-G, V-MoE, and CoAtNet-7 pushed vision models toward billion-scale capacity, but largely relied on JFT-3B and image classification. A general large vision model still needed stable deep optimization, reliable transfer from low to high resolution, and a less label-intensive training recipe.
Core Problem
In pre-normalized residual blocks, branch outputs are added directly to the main stream, causing activation amplitudes to grow with depth. The observed discrepancy can reach 10^4, and a 658M-parameter model may fail before training finishes. Fixed relative-bias tables cover only one window range; bicubic interpolation is ad hoc under large resolution changes. High-resolution attention also creates severe memory pressure from activations, optimizer states, and parameters.
Innovation
First, residual post-normalization normalizes residual outputs before addition, while scaled cosine attention uses cos(q,k)/τ with head- and layer-specific τ. Second, Log-CPB generates B(Δx,Δy) through a small MLP and feeds it logarithmic coordinates, reducing extrapolation when windows grow. Third, SimMIM reduces label dependence. ZeRO stage-1, activation checkpointing, and sequential self-attention make these ideas executable on 40GB GPUs rather than purely theoretical.
Methodology
- �� Backbone: retain Swin's four-stage hierarchy, local windows, and shifted windows.
- �� Stabilization: move LayerNorm to each residual branch output; add another main-branch normalization every six blocks in H/G.
- �� Attention: compute logits as cos(q_i,k_j)/τ+B_ij, with τ>0.01 and not shared across heads or layers.
- �� Position transfer: use a two-layer ReLU MLP G over sign(x)log(1+|x|) coordinates.
- �� Pre-training: train SwinV2-G with SimMIM for 20 epochs on ImageNet-22K-ext, then classification pre-train for 30 epochs.
- �� Memory: partition states with ZeRO, recompute activations when needed, and sequentially compute early-stage attention.
Experiments
Evaluation covers ImageNet-1K V1/V2, COCO, ADE20K, and Kinetics-400. Pre-training uses ImageNet-22K, private ImageNet-22K-ext, and Object365 v2 for an intermediate detection phase. Models range from SwinV2-T/S/B/L to 658M H and 3B G. Ablations compare parameterized bias, linear-spaced CPB, and Log-CPB across window sizes, including transfer from 8×8 to much larger windows.
Results
SwinV2-G obtains 84.00% on ImageNet-V2 and 90.17% on ImageNet-V1. It reaches 63.1 box AP and 54.4 mask AP on COCO, 59.9 mIoU on ADE20K, and 86.8% top-1 on K400. Relative to the best original Swin numbers, the reported gains are about 0.8, 1.8/1.4, 1.5, and 1.4 points. The Swin-T ablation shows that Log-CPB is particularly valuable as window size increases.
Applications
The backbone can support high-resolution detection, instance and semantic segmentation, and video recognition. Its 1536×1536 COCO setting is relevant to remote sensing, industrial inspection, medical imaging, and autonomous driving, where small objects matter. Kinetics-400 indicates transfer to spatiotemporal recognition. Deployment will likely require distillation, pruning, quantization, or smaller SwinV2 variants.
Limitations & Outlook
Exact reproduction requires a private 70M-image corpus and a large A100 cluster. High-resolution training remains expensive, and checkpointing can slow training by up to 30%. The paper does not systematically report inference cost, energy, robustness, fairness, or zero-shot behavior. Future work should use open data, evaluate parameter-efficient adaptation, explore sparsity and multimodal training, and measure scaling under matched computational budgets.
Plain Language Accessible to non-experts
Imagine a factory that must inspect products of many sizes. The older factory works well on small items, but as more floors are added, vibrations accumulate and the upper floors become unstable. Residual post-normalization acts like a shock absorber at the exit of every workshop, preventing small disturbances from growing into a collapse. Cosine attention is like first adjusting every worker's speaking volume before deciding whose message matters, so one unusually loud worker cannot dominate the whole meeting.
Now imagine that the factory learned using small boxes but must handle huge crates. Stretching the old layout is crude: distances and locations become distorted. Log-CPB is a flexible map that calculates relationships from distance, using a gentler scale for faraway places. It can therefore produce sensible layouts for new window sizes.
Finally, SimMIM lets the factory learn from partly covered product photos by guessing what is missing, instead of requiring a human label for every image. With memory-saving software, the factory grows to 3B adjustable parts and performs strongly on classification, detection, segmentation, and video tasks. The trade-off is that such a factory still needs considerable electricity, hardware, and carefully prepared data.
ELI14 Explained like you're 14
Think of a game character that must recognize objects, enemies, and movements across changing maps. A normal character may do fine in a small room, but on a giant map it can lose track of things or become unstable. Swin V2 upgrades the character: residual post-normalization is like checking the compass after every level so mistakes do not pile up; cosine attention is like turning everyone’s microphone to a similar volume before deciding who is saying something important.
What if the character trained on tiny rooms but is suddenly sent into a huge arena? Old systems stretch their old map and hope for the best. Log-CPB teaches the character how relationships change with distance, so it can handle larger windows more naturally.
Training is also expensive. SimMIM hides parts of pictures and asks the model to guess them, like solving a puzzle without a teacher writing an answer on every piece. ZeRO and checkpointing help the model fit into limited GPU memory.
The result is a giant 3B-parameter model: 84.0% on ImageNet-V2, 63.1 box AP on COCO, 59.9 mIoU on ADE20K, and 86.8% on Kinetics-400. That is impressive! But it needs powerful computers and lots of data, so researchers still need smaller versions for phones, robots, and everyday products.
Glossary
Swin Transformer
A hierarchical vision Transformer that computes attention within local windows and shifts those windows between blocks. This combines efficient computation with cross-window communication.
Swin V2 preserves this backbone while changing normalization, attention, and positional bias.
Residual Post-Norm
LayerNorm is applied after a residual branch produces its output and before that output is merged into the main stream. The design reduces depth-dependent activation growth.
It replaces the pre-normalization configuration used by Swin V1.
Scaled Cosine Attention
Attention logits use cosine similarity divided by a learnable temperature τ rather than an unnormalized dot product. This makes attention less sensitive to vector magnitude.
It is introduced to stabilize large vision models.
Log-CPB
A continuous relative position bias generated by an MLP from logarithmically transformed coordinates. It supports arbitrary window sizes with reduced extrapolation.
It replaces fixed bias tables and bicubic interpolation.
SimMIM
A self-supervised method that masks image regions and reconstructs their contents. It learns visual representations without requiring a class label for every image.
SwinV2-G uses SimMIM for its first pre-training phase.
ZeRO
A memory optimization method that partitions model states and optimizer states across GPUs. It reduces redundant storage in distributed training.
The experiments use DeepSpeed ZeRO stage-1.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether a 3B model retains its gains on fully open and much smaller datasets; matched-data and matched-compute studies are needed for reproducibility.
- 2 Log-CPB has not been thoroughly tested on long videos, 3D point clouds, or irregular grids, where spatial coordinates and efficiency differ from 2D images.
- 3 Accuracy is well documented, but energy, inference latency, dataset bias, robustness, and zero-shot transfer receive limited analysis; a broader evaluation is needed.
Applications
Immediate Applications
High-resolution industrial inspection
Manufacturers can use SwinV2 as a detection or segmentation backbone for tiny defects in large product images. Domain data, calibration, and deployment compression such as distillation or quantization are needed to control the 3B model's cost.
Remote sensing and medical imaging
Building extraction, lesion segmentation, and similar tasks benefit from large inputs and fine spatial detail. Log-CPB can ease transfer to larger windows, but privacy, domain validation, and professional certification remain prerequisites.
Long-term Vision
General vision-language foundation models
SwinV2 could serve as a high-capacity visual encoder for language, video, robotics, and embodied systems. Major obstacles include open multimodal data, alignment quality, energy consumption, safety, and efficient adaptation.
Abstract
Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities like that of human beings. This paper aims to explore large-scale models in computer vision. We tackle three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger on labelled data. Three main techniques are proposed: 1) a residual-post-norm method combined with cosine attention to improve training stability; 2) A log-spaced continuous position bias method to effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; 3) A self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Through these techniques, this paper successfully trained a 3 billion-parameter Swin Transformer V2 model, which is the largest dense vision model to date, and makes it capable of training with images of up to 1,536$\times$1,536 resolution. It set new performance records on 4 representative vision tasks, including ImageNet-V2 image classification, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 video action classification. Also note our training is much more efficient than that in Google's billion-level visual models, which consumes 40 times less labelled data and 40 times less training time. Code is available at \url{https://github.com/microsoft/Swin-Transformer}.