Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
SETR uses a pure Transformer for segmentation, reaching 50.28% mIoU on ADE20K.
Key Findings
Methodology
The paper reframes semantic segmentation as sequence-to-sequence prediction. An image is split into 16×16 patches, each patch is linearly embedded into a token, and a pure Transformer encoder—without convolution or spatial downsampling—models global context at every layer. The encoder output is reshaped back to a 2D feature map and passed to simple decoders. Three decoder variants are studied: Naive, Progressive Upsampling (PUP), and Multi-Level Aggregation (MLA), isolating the representational power of the Transformer encoder.
Key Results
- On ADE20K, SETR-MLA reaches 50.28% mIoU with multi-scale testing after 160k iterations, outperforming FCN (41.40%) and DeepLabV3+ (46.40%) by a wide margin; SETR-PUP also scores 50.09%.
- On Pascal Context, SETR-MLA achieves 55.83% mIoU, ahead of APCNet (54.70%), Strip Pooling (54.50%), EMANet (53.10%), and DANet (52.60%). SETR-PUP follows closely at 55.27%.
- On Cityscapes, SETR-PUP-DeiT-Base attains 79.45% mIoU, surpassing FCN (76.61%), PSPNet (78.50%), and DeepLab-v3 (79.30%), showing that a pure Transformer backbone is competitive on high-resolution street scenes.
Significance
This work matters because it breaks the long-standing assumption that semantic segmentation must rely on a convolutional encoder-decoder pipeline. Conventional FCNs grow receptive fields gradually and often need dilation or attention add-ons to model long-range context. SETR instead gives every layer a global receptive field from the start, preserving a uniform token representation and avoiding early spatial collapse. For research, it establishes segmentation as a true sequence modeling problem; for deployment, it suggests a cleaner, more transferable vision backbone that can be reused across tasks.
Technical Contribution
The technical contribution has three pillars. First, SETR replaces stacked convolutions with a pure Transformer encoder, using patch tokenization, positional embeddings, and standard multi-head self-attention defined by Q=ZW_Q, K=ZW_K, V=ZW_V, followed by softmax(QK^T/√d)V. Second, it proposes three decoders—Naive one-shot upsampling, PUP staged 2× upsampling, and MLA multi-layer fusion—to probe how much segmentation quality comes from the encoder versus the decoder. Third, it demonstrates strong transfer from ViT/DeiT pretraining on Cityscapes, ADE20K, and Pascal Context, including state-of-the-art results on ADE20K at submission time.
Novelty
The novelty is not merely “adding attention” to FCN; it removes convolutional downsampling altogether and keeps the representation sequence-like throughout. Unlike Non-Local, DANet, CCNet, or Axial-Attention, which still sit on top of convolutional backbones or hierarchical feature pyramids, SETR makes the Transformer the sole encoder. That makes it one of the first works to show that dense prediction can be solved effectively with a pure Transformer backbone rather than a CNN-plus-attention hybrid.
Limitations
- The model is computationally heavy: even with 16×16 patches, T-Large has 305.67M parameters, and global self-attention scales quadratically with token count, limiting direct extension to finer patches or much larger inputs.
- Performance depends strongly on large-scale pretraining. Random initialization is substantially worse, e.g., SETR-PUP-R on Cityscapes reaches only 42.27% mIoU, indicating that optimization and generalization are fragile without ViT/DeiT-style pretraining.
Future Work
Likely follow-up directions include more efficient attention mechanisms, hierarchical token representations, and lighter decoders to reduce memory cost. The paper also motivates extending pure-Transformer segmentation to weakly supervised learning, video segmentation, and remote sensing, where global context is crucial but compute efficiency remains a bottleneck.
AI Executive Summary
Semantic segmentation has long been dominated by FCN-style encoder-decoder systems: a convolutional encoder compresses the image, enlarges the receptive field, and a decoder restores pixel-level predictions. That recipe is effective, but it carries a structural weakness. Long-range dependencies are still hard to capture, and many recent improvements—dilated convolutions, pyramid pooling, and attention modules—mostly patch the same basic architecture rather than replacing it.
SETR proposes a different worldview: treat semantic segmentation as sequence-to-sequence learning. The image is split into 16×16 patches, each patch is flattened and linearly projected into a token embedding, and positional embeddings are added before feeding the sequence into a pure Transformer encoder. No convolutional backbone, no progressive resolution reduction. Instead, every Transformer layer performs global self-attention, so any patch can directly interact with every other patch through the standard Q/K/V mechanism. To recover the full-resolution segmentation map, the authors test three decoders: Naive one-shot upsampling, PUP progressive upsampling, and MLA multi-level aggregation.
The result is a striking empirical validation of the idea. With ViT/DeiT pretraining, SETR achieves 50.28% mIoU on ADE20K, 55.83% on Pascal Context, and 79.45% on Cityscapes with the PUP-DeiT-Base variant. These numbers surpass strong FCN-family baselines such as FCN, PSPNet, DeepLab-v3, and several attention-enhanced models. Just as importantly, the study shows that a simple decoder is enough when the encoder learns rich global representations. SETR therefore does more than improve a benchmark: it reframes dense prediction as a problem of global sequence modeling, opening a path toward a unified Transformer-based vision stack.
Deep Analysis
Background
Semantic segmentation evolved from FCN to encoder-decoder architectures because pixel-wise classification requires both semantic abstraction and spatial detail. Architectures such as FCN, U-Net-like designs, DeepLab, PSPNet, and later attention-enhanced models like PSANet, DANet, CCNet, and OCRNet all aimed to enlarge the receptive field or enrich context modeling. However, most still rely on convolutional backbones such as VGG or ResNet, which learn features locally and only gradually expand context. In parallel, Transformer models transformed NLP and machine translation, and ViT/DeiT showed that pure Transformers can also work for image classification. SETR imports that paradigm into dense prediction.
Core Problem
The paper targets a precise bottleneck: how to model long-range context for semantic segmentation without losing spatial fidelity. In FCNs, downsampling is used to increase semantic abstraction, but it also discards fine-grained location cues. Attention modules can help, yet they are often applied only at higher, smaller-resolution stages because of quadratic cost. This creates a tension: the lower layers, where detailed localization lives, are under-modelled; the higher layers, where context is available, are too coarse. The challenge is therefore to design an encoder that is simultaneously global, spatially aware, and computationally practical.
Innovation
SETR introduces several tightly coupled innovations. • Pure Transformer encoder: replaces the convolutional backbone with a patch-token sequence encoder, so global context is available in every layer. • Patch-based input representation: converts H×W×3 images into HW/256 tokens using 16×16 patches, avoiding pixel-level token explosion. • Learned positional embeddings: preserve 2D location information despite the permutation-invariant nature of self-attention. • Decoder ablation suite: Naive, PUP, and MLA decoders separately test whether segmentation quality comes from encoder features or decoder sophistication. • Multi-layer supervision: auxiliary losses on intermediate Transformer layers improve optimization, echoing deep supervision in FCNs while staying within the Transformer framework. • Pretraining transfer: ViT and DeiT weights are interpolated to new input sizes, enabling effective fine-tuning.
Methodology
- �� Input tokenization: an image x∈R^{H×W×3} is partitioned into a regular grid of 16×16 patches, each patch flattened and linearly projected to a C-dimensional embedding.
- �� Position encoding: a learned positional vector p_i is added to each patch embedding e_i, forming E={e_i+p_i}. This keeps absolute spatial information after tokenization.
- �� Transformer encoder: the sequence is processed by Le stacked Transformer layers. Each layer uses LayerNorm, multi-head self-attention, and an MLP block with residual connections. For one head, attention is computed as SA(Z)=Z+softmax(ZW_Q(ZW_K)^T/√d)(ZW_V), where W_Q, W_K, W_V are learned projections.
- �� Output reshaping: the final sequence Z is reshaped from H/16×W/16×C back into a 2D feature map for decoding.
- �� Naive decoder: a 2-layer 1×1 conv head maps features to class logits and bilinearly upsamples once to the original resolution. It is the simplest test of encoder quality.
- �� PUP decoder: uses repeated 2× upsampling interleaved with convolutions to reduce artifacts and produce smoother boundaries.
- �� MLA decoder: extracts features from multiple Transformer depths (e.g., Z_6, Z_12, Z_18, Z_24), processes each stream with 1×1/3×3/3×3 convolutions, fuses them top-down by element-wise addition, and concatenates the streams before a final 4× upsample. This is FPN-like in spirit, but unlike FPN it does not rely on a hierarchical pyramid of feature resolutions.
Experiments
The evaluation covers three standard segmentation benchmarks: Cityscapes (5000 finely annotated urban street scenes, 19 classes), ADE20K (20210/2000/3352 train/val/test images, 150 classes), and PASCAL Context (59 frequent classes plus background, 60 classes total). Training follows mmsegmentation defaults: random resize in [0.5, 2], random cropping, and horizontal flipping. The optimizer is SGD with polynomial learning-rate decay; batch size is 16 for ADE20K and Pascal Context, 8 for Cityscapes. The authors compare against FCN, Semantic FPN, DeepLab variants, PSPNet, DANet, OCRNet, UperNet, CCNet, and other context-modeling baselines. They report both single-scale and multi-scale inference, and test T-Base versus T-Large encoders, plus ViT/DeiT pretraining versus random initialization.
Results
The key message is that SETR is consistently competitive or state of the art. On ADE20K, SETR-MLA reaches 50.28% mIoU with multi-scale testing, versus 46.40% for DeepLabV3+ and 41.40% for FCN under the reported settings. On Pascal Context, SETR-MLA reaches 55.83%, beating APCNet (54.70%), Strip Pooling (54.50%), and EMANet (53.10%). On Cityscapes, SETR-PUP-DeiT-Base achieves 79.45%, ahead of FCN (76.61%), PSPNet (78.50%), and DeepLab-v3 (79.30%). The ablations also reveal a clear pattern: PUP generally improves over Naive, and MLA is particularly effective on complex scene parsing datasets. Randomly initialized transformers perform much worse, underscoring the importance of large-scale pretraining.
Applications
SETR is directly relevant to any dense prediction problem where global scene context matters: autonomous driving, aerial and satellite imagery, medical image parsing, and indoor scene understanding. In practice, teams could adopt the patch-token encoder and swap only the decoder head for their label set. The model is especially attractive when strong pretraining is available and accuracy matters more than latency. It also offers a cleaner conceptual foundation for systems that may later be extended to detection or panoptic perception.
Limitations & Outlook
The main limitation is efficiency. Global self-attention scales quadratically with the number of tokens, so even a 16×16 patch size already pushes memory and compute limits; the T-Large model is 305.67M parameters. A second limitation is data dependence: the model benefits strongly from ViT/DeiT pretraining, and performance drops sharply when initialized from scratch. A third concern is boundary precision under the simplest decoder, where one-shot upsampling can produce noisier outputs than progressive or multi-level decoding. These issues motivate more efficient attention and stronger decoder design.
Plain Language Accessible to non-experts
Imagine a huge photo printed on a poster. The old way of solving segmentation is a bit like shrinking that poster again and again until it becomes a tiny postcard, then trying to guess what every spot on the original poster was supposed to be. That works okay for big obvious things, but tiny details get blurred out. SETR takes a different route: it cuts the poster into many equally sized squares, like pieces of a puzzle, and keeps each piece in a form the computer can read.
Now imagine a very smart team leader who stands above the puzzle table. Instead of looking at only one nearby piece, the leader can look at all the pieces at once and notice how they relate to each other. A road piece might connect to a car piece; a sky piece might sit above a building piece. Because every piece can “talk” to every other piece, the system doesn’t need to keep shrinking the picture to understand the scene.
At the end, the leader helps put the puzzle back together into a full map where every tiny dot has a label. The paper even tries three different ways of putting the puzzle back: all at once, step by step, and by combining several rounds of thinking. The best version gets very strong results on difficult datasets like ADE20K, Pascal Context, and Cityscapes. So the big idea is simple: don’t force the picture to become tiny before understanding it. Let the whole picture stay visible, and learn the relationships directly.
ELI14 Explained like you're 14
Think about how you recognize a scene in a game or on social media. You don’t zoom in on just one tiny corner and hope to figure out the whole picture, right? You look at the whole screen, notice the road, the sky, the people, and where everything sits. SETR does something like that for computer vision.
Old segmentation models are a bit like a person who keeps folding a map smaller and smaller to save space. Sure, the map becomes easier to carry, but details disappear! SETR says: “Why fold it so much?” Instead, it cuts the image into little blocks and lets a Transformer watch all the blocks together. That means every block can pay attention to every other block. Cool, right?
Then comes the “put it back together” part. The paper tests three ways to rebuild the final label map. One is simple and fast, one is careful and step-by-step, and one mixes information from several layers. The careful versions work best. On hard benchmarks, the results are impressive: 50.28% mIoU on ADE20K, 55.83% on Pascal Context, and 79.45% on Cityscapes. Those are not small wins!
So what’s the big lesson? Sometimes the best way to understand a picture is not to squish it into a tiny version first. Instead, keep the whole scene in view and let the model connect the dots itself. That’s exactly what SETR does, and that’s why people cared so much about it.
Glossary
Transformer
A model built around self-attention rather than convolutions. In plain terms, it lets every token compare itself with all other tokens; technically, it learns global dependencies through Q/K/V projections and attention weights.
SETR uses a pure Transformer as the encoder, replacing the conventional CNN backbone.
Patch Embedding
A way to turn an image into a sequence by slicing it into fixed-size patches and mapping each patch to a vector. This makes the image compatible with sequence models.
SETR uses 16×16 patches to create the input token sequence.
Multi-Head Self-Attention
Several attention operations run in parallel, each focusing on different relations. Their outputs are concatenated and projected back to the hidden dimension.
This is the main mechanism that gives SETR global context at every layer.
mIoU
Mean Intersection over Union, the standard segmentation metric. It measures how much predicted regions overlap with ground-truth regions, averaged over classes.
All major results in the paper are reported using mIoU on Cityscapes, ADE20K, and Pascal Context.
Decoder
The module that turns high-level features into a pixel-wise class map. In simple terms, it restores the original image resolution for segmentation.
SETR studies Naive, PUP, and MLA decoder variants.
Open Questions Unanswered questions from this research
- 1 The paper shows that pure Transformers work well, but it does not settle the best token granularity or whether a hierarchical token structure would be better for even larger images. How to keep global context while cutting memory cost remains open.
- 2 SETR relies heavily on large-scale pretraining. It remains unclear how robust the approach is under limited labels, heavy domain shift, or noisy annotations, especially in specialized fields like medical or remote-sensing segmentation.
Applications
Immediate Applications
Autonomous driving scene parsing
Useful for identifying roads, vehicles, pedestrians, and lane-adjacent structures at pixel level. Teams with strong GPUs and pretrained weights can integrate it into perception stacks for planning and safety monitoring.
High-accuracy scene understanding
Applicable to remote sensing, indoor navigation, and medical image parsing, where global context matters. It can be adapted by replacing the label head while keeping the same tokenized encoder.
Long-term Vision
Unified visual sequence modeling
A longer-term vision is that segmentation, detection, depth estimation, and other dense vision tasks may share a common token-based Transformer backbone. The main barrier is efficiency, but the conceptual payoff is a more unified vision architecture.
Abstract
Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (ie, without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.