Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network
Proposed Global Convolutional Network (GCN) improves semantic segmentation, achieving 82.2% on VOC2012 and 76.9% on Cityscapes.
Key Findings
Methodology
The GCN integrates symmetric separable large kernels to enlarge receptive fields, enhancing both classification and localization. It employs parameter-efficient large kernels (k=15) without nonlinearities, combined with multi-scale feature fusion and end-to-end training. A residual boundary refinement module further improves boundary accuracy. Experiments on VOC2012 and Cityscapes validate the effectiveness of large kernels, showing significant performance gains, especially in internal regions, with boundary details refined via residual modules.
Key Results
- On VOC2012 validation, GCN achieves 82.2% mIoU, surpassing previous best of 80.2%. On Cityscapes, 76.9% mIoU was obtained, outperforming prior methods. Kernel size k=15 consistently outperformed smaller kernels, with a 5.5% gain over small kernel baselines. Boundary refinement further improved boundary pixel accuracy, boosting overall performance.
- Ablation studies confirmed large kernels (k=15) outperform stacked small kernels and trivial large kernels, with better parameter efficiency and training stability. The pre-trained ResNet-GCN model showed improved transfer learning results, validating the architecture’s robustness across datasets and tasks.
- Multi-scale feature fusion and CRF post-processing further enhanced results, demonstrating the synergy of global receptive fields and local boundary refinement. The model’s ability to handle large objects and boundary details was quantitatively verified through internal and boundary region analysis.
Significance
This work addresses the fundamental contradiction in semantic segmentation between classification invariance and localization sensitivity. By introducing large kernels, the model captures extensive global context, significantly improving pixel-wise accuracy. The end-to-end boundary refinement complements this by sharpening object edges. These innovations push the state-of-the-art, offering a practical, scalable solution for high-precision scene understanding, with broad implications for autonomous driving, robotics, and scene analysis.
Technical Contribution
The paper introduces the symmetric separable large kernel convolution (GCN), which maintains parameter efficiency while enlarging receptive fields. It innovates with a residual boundary refinement module integrated into the network, enabling precise boundary localization. The combination of multi-scale fusion, large kernels, and boundary refinement results in a unified, end-to-end trainable architecture that surpasses existing methods in accuracy and robustness.
Novelty
This is the first systematic application of symmetric separable large kernels in semantic segmentation, effectively resolving the receptive field versus parameter trade-off. The boundary residual refinement module is also novel, directly integrated into the network for end-to-end optimization. Compared to prior works relying on stacking small kernels or post-processing CRFs, this approach offers a more elegant, efficient solution with superior performance.
Limitations
- Despite improvements, boundary refinement remains less effective in extremely cluttered or occluded scenes. Large kernels increase computational load, especially for very large values (k>15). The model’s generalization to different domains or real-time applications needs further validation. Future work should explore adaptive kernel sizes and more efficient boundary modeling techniques.
Future Work
Future directions include developing adaptive kernel mechanisms for scene-specific receptive fields, integrating attention modules to enhance global context understanding, and optimizing the architecture for real-time deployment. Extending the approach to 3D data and video segmentation, as well as exploring unsupervised or semi-supervised training paradigms, are promising avenues.
AI Executive Summary
Semantic segmentation remains a challenging task in computer vision, requiring precise pixel-wise classification and localization. Traditional methods often struggle to balance these conflicting demands, especially when handling large objects or complex boundaries. Recent approaches leverage deep CNNs like FCN, DeepLab, and ResNet, but are limited by receptive field constraints and local feature focus. To address this, the present work introduces the Global Convolutional Network (GCN), which employs symmetric separable large kernels (k=15) to significantly expand the receptive field without excessive parameter growth. This design enables the network to incorporate global context, improving classification robustness across transformations.
The architecture combines multi-scale feature fusion, deconvolution-based upsampling, and a residual boundary refinement module, all trained end-to-end. Extensive experiments on VOC2012 and Cityscapes demonstrate that GCN achieves state-of-the-art results, with 82.2% and 76.9% mean IoU respectively. Ablation studies confirm that large kernels outperform stacking small kernels or trivial large kernels, with a notable 5.5% gain. Boundary refinement further enhances edge accuracy, especially near object borders.
This approach addresses the core challenge of balancing classification invariance with localization sensitivity, offering a scalable, efficient solution for high-precision scene understanding. Its success paves the way for future research into adaptive receptive fields, attention mechanisms, and real-time applications, promising broad impact in autonomous driving, robotics, and beyond.
Deep Analysis
Background
语义分割作为计算机视觉中的核心任务，经历了从传统图像处理到深度学习的快速演变。早期方法依赖手工特征，效果有限。深度卷积网络如FCN、DeepLab系列极大推动了性能提升，尤其是端到端训练、多尺度融合和空洞卷积等技术，但仍存在感受野不足、边界模糊等难题。现有模型多偏重局部特征，难以捕获全局信息，限制了复杂场景中的表现。近年来，研究者开始关注扩大感受野和细节捕获，尝试多尺度融合、全局池化等手段，但仍未解决分类与定位的矛盾。
Core Problem
核心问题在于如何在保持像素级分类准确的同时，实现精确的空间定位。传统模型多采用局部连接，导致对大尺度对象识别不足，边界细节模糊。堆叠小核卷积虽参数高效，但感受野有限，难以捕获全局信息，影响模型鲁棒性。如何设计一种既能扩大感受野，又参数控制合理的架构，成为研究难点。本文提出大核卷积，旨在突破这一瓶颈，提升模型整体性能。
Innovation
创新点一：引入对称分离大核卷积（k=15），在保持参数效率的同时，显著扩大感受野，增强全局感知。创新点二：结合残差边界细化模块，改善边界区域像素的准确性。创新点三：采用多尺度特征融合与反卷积上采样，提升像素级细节表现。相较于堆叠小核卷积或全局池化，GCN在参数效率和性能上实现突破，兼顾分类与定位，解决边界模糊和尺度变化问题。
Methodology
- �� 输入：利用预训练ResNet提取多尺度特征。• GCN模块：采用对称分离的k×k卷积（如k=15），实现大感受野，参数控制合理。• 特征融合：多尺度特征融合，通过逐层反卷积上采样，生成高分辨率分割图。• 边界细化：引入残差边界细化模块，优化边界对齐，端到端训练。• 损失函数：结合多尺度交叉熵和边界误差，确保像素级精度。• 训练策略：在VOC和Cityscapes上微调，使用数据增强和优化技巧，确保模型稳定收敛。
Experiments
在VOC2012和Cityscapes上进行消融实验，验证不同核大小（k=3到15）对性能的影响。比较GCN与普通堆叠卷积、全局卷积和简单大核卷积的效果。采用平均IoU指标，分析边界和内部区域的性能差异。验证预训练ResNet-GCN的迁移能力，结合多尺度融合和CRF后处理，性能持续提升。实验还包括参数量、训练稳定性和边界细节改善的分析。
Results
大核卷积（k=15）在VOC2012达82.2%，Cityscapes达76.9%，显著优于小核和堆叠卷积。边界细化模块提升边界像素准确率，整体性能增强。消融结果显示，大核卷积在参数控制和训练稳定性方面优越。预训练模型迁移效果明显，验证架构的实用性。多尺度融合和CRF后处理进一步提升性能，验证了全局感知与边界优化的有效结合。
Applications
该模型适用于自动驾驶、场景理解、机器人视觉等领域，能实现高精度像素级识别。只需微调预训练模型，即可部署于实际场景中。模型对大尺度对象识别和边界细节优化具有明显优势，适合复杂环境的实时应用，推动智能视觉技术发展。
Limitations & Outlook
模型在极端尺度变化或复杂背景中仍存在边界模糊问题。大核卷积增加计算成本，超大核（k>15）训练难度上升。泛化能力在不同任务和数据集上尚需验证，未来需结合更高效的特征编码机制以提升实用性。
Plain Language Accessible to non-experts
想象你在厨房做菜，用大锅炖汤，能一次性煮更多食材，但需要掌握火候，否则容易糊锅。传统用小刀切菜，细节多但处理慢。大核卷积就像用大锅，能一次性覆盖更多“区域”，让“厨艺”更高效。边界细化像调味料，让菜看起来更漂亮、更有层次。这样，电脑“看”图片时，也能像厨师一样，把大局和细节都照顾到，做出更美味的“菜”。
ELI14 Explained like you're 14
你知道吗？让电脑理解图片里的东西，有两个难题：一是要知道每个部分是什么（分类），二是要知道它们在哪（定位）。以前的方法像用小刀切菜，虽然方便，但处理大块食材很慢，也不够全。这个新方法像用大锅炖汤，能一次性照顾到更多区域，让电脑更聪明。它还用特别的“调味料”让边界更清楚，看得更细。结果显示，这样的做法让电脑在识别图片中的物体时，比以前更准确、更细腻，特别是在复杂的场景中表现出色。未来，这种技术还能帮自动驾驶汽车更好地“看清”路上的东西，变得更安全、更智能！
Abstract
One of recent trends [30, 31, 14] in network architec- ture design is stacking small filters (e.g., 1x1 or 3x3) in the entire network because the stacked small filters is more ef- ficient than a large kernel, given the same computational complexity. However, in the field of semantic segmenta- tion, where we need to perform dense per-pixel prediction, we find that the large kernel (and effective receptive field) plays an important role when we have to perform the clas- sification and localization tasks simultaneously. Following our design principle, we propose a Global Convolutional Network to address both the classification and localization issues for the semantic segmentation. We also suggest a residual-based boundary refinement to further refine the ob- ject boundaries. Our approach achieves state-of-art perfor- mance on two public benchmarks and significantly outper- forms previous results, 82.2% (vs 80.2%) on PASCAL VOC 2012 dataset and 76.9% (vs 71.8%) on Cityscapes dataset.