Learning Enriched Features for Fast Image Restoration and Enhancement
MIRNet-v2 employs multi-scale residual blocks with attention mechanisms, achieving state-of-the-art results in image restoration tasks, reducing parameters by 81%.
Key Findings
Methodology
The approach centers on multi-scale residual blocks (MRB) that integrate parallel convolution streams at various resolutions, enabling rich feature extraction. These streams exchange information through a progressive fusion process enhanced by a selective kernel fusion (SKFF) module, which dynamically adjusts receptive fields via self-attention. Residual contextual blocks (RCB) further refine features by modeling inter-channel dependencies with group convolutions and attention. The architecture employs a recursive residual design, maintaining high-resolution details while aggregating contextual cues from lower resolutions. Training utilizes Charbonnier loss, with progressive patch sizes to improve efficiency.
Key Results
- On six real datasets, MIRNet-v2 outperforms existing SOTA methods, achieving a PSNR of 39.84dB on SIDD, surpassing previous 39.72dB, with 81% fewer parameters and 3.6× faster inference. On DPDD, PSNR reaches 28.96dB, significantly better than prior approaches. Ablation studies confirm the effectiveness of multi-scale exchange and SKFF, with notable improvements in detail preservation and robustness across tasks.
- In denoising, MIRNet-v2 exceeds CycleISP and DAGL, with gains of 0.32dB and 0.11dB PSNR respectively on SIDD and DND, demonstrating strong generalization. The model maintains high performance across diverse noise levels and scene complexities, validating its multi-task capability.
- Ablation results reveal that progressive feature exchange and adaptive kernel fusion are crucial for performance gains, reducing model complexity without sacrificing accuracy. The architecture’s robustness across tasks and datasets underscores its potential for real-world applications.
Significance
This work addresses the longstanding challenge of balancing spatial detail preservation with global contextual understanding in image restoration. By innovatively combining multi-scale residual learning with dynamic receptive field adjustment, it sets a new benchmark in both accuracy and efficiency. Its ability to perform multiple tasks with a unified architecture accelerates deployment in practical scenarios like mobile photography, autonomous systems, and remote sensing, where high-quality image recovery is critical. The approach also opens avenues for further research into adaptive multi-scale feature processing and efficient neural network design.
Technical Contribution
The core technical contributions include the design of multi-scale residual blocks that maintain high-resolution details, the introduction of SKFF for dynamic feature fusion based on self-attention, and the integration of residual contextual blocks for enhanced feature modeling. These innovations collectively reduce computational costs while boosting performance. The recursive residual structure ensures efficient training and better gradient flow. The architecture’s modularity facilitates multi-task learning, setting a new standard for scalable, high-performance image restoration models.
Novelty
This research uniquely combines multi-scale residual learning with a dynamic kernel fusion mechanism, enabling the network to adaptively select receptive fields for each feature map. Unlike prior multi-scale methods that rely on simple concatenation or fixed fusion, MIRNet-v2’s SKFF module dynamically recalibrates features, leading to superior detail preservation and contextual understanding. Its recursive residual design and residual contextual blocks further distinguish it from existing architectures, making it a pioneering approach in multi-task image restoration.
Limitations
- Despite its efficiency, the model’s complexity still poses challenges for deployment on extremely resource-constrained devices. Handling ultra-high-resolution images or real-time video remains computationally demanding.
- The model’s performance may degrade in scenarios with extremely severe noise or artifacts beyond the training distribution, indicating a need for broader training datasets.
- Further research is needed to optimize the architecture for real-time applications and to extend its capabilities to video and 3D scene restoration.
Future Work
Future directions include developing lightweight variants for edge devices, integrating self-supervised learning to enhance generalization, and extending the architecture to video and 3D data. Exploring unsupervised or semi-supervised training could reduce reliance on large labeled datasets. Additionally, combining this approach with generative models may further improve restoration quality in challenging scenarios.
AI Executive Summary
Image restoration remains a fundamental challenge in computer vision, vital for applications ranging from photography to autonomous driving. Traditional CNN-based methods often face a trade-off: encoder-decoder architectures efficiently capture global context but lose spatial details, while high-resolution networks preserve details but struggle with contextual understanding. To address this, the paper introduces MIRNet-v2, a novel architecture that harmonizes high-resolution detail preservation with rich multi-scale contextual encoding.
Central to MIRNet-v2 are multi-scale residual blocks (MRB), which process features in parallel at various resolutions. These streams exchange information through a progressive fusion process, enhanced by a selective kernel fusion (SKFF) module that dynamically adjusts receptive fields based on self-attention. Residual contextual blocks (RCB) further refine features by modeling inter-channel dependencies, ensuring the network captures both local textures and global semantics effectively.
The architecture employs a recursive residual design, maintaining high-resolution details throughout the network while aggregating contextual cues from coarser scales. Extensive experiments on six real-world datasets demonstrate MIRNet-v2’s superior performance across tasks such as defocus deblurring, denoising, super-resolution, and enhancement. For instance, on the SIDD dataset, it achieves a PSNR of 39.84dB, surpassing previous methods, with parameters reduced by 81% and inference speed increased by 3.6 times.
This work significantly advances the state-of-the-art by providing a scalable, efficient, and versatile framework for multi-task image restoration. Its ability to balance detail preservation with contextual understanding addresses core limitations of prior models, paving the way for practical deployment in real-world scenarios like mobile photography and autonomous systems. Future research will focus on model lightweighting, unsupervised training, and extension to video and 3D scene restoration, promising broader impact and continued innovation in the field.
Deep Dive
Glossary
Residual Learning (残差学习)
A technique where the network learns the difference between input and target, facilitating training of deeper models; in Chinese: 学习输入与目标的差异以简化训练。
用于模型架构中以增强深层网络的训练效果。
Self-Attention (自注意力)
机制让模型在特征融合时考虑不同位置的重要性，从而增强表达能力；在论文中用于动态调节感受野。
在SKFF模块中实现特征的自适应调节。
Receptive Field (感受野)
神经网络中单个神经元能感知的输入区域大小，影响模型对上下文的理解能力；在论文中动态调节以融合多尺度信息。
通过SKFF机制调节感受野大小。
Group Convolution (组卷积)
一种卷积操作，将输入通道分组，减少参数量，提高效率；在RCB中用于特征提取。
提升模型效率同时增强特征表达。
Charbonnier Loss (夏尔博尼损失)
一种平滑的L1损失，减少训练中的不稳定性，优化细节恢复；在训练中使用以提升性能。
作为模型的损失函数。
Open Questions Unanswered questions from this research
- 1 尽管模型在多任务中表现优异，但在极端噪声或模糊场景下的鲁棒性仍需提升，特别是在超高分辨率或实时应用中。
- 2 模型复杂度较高，硬件资源需求大，限制了在低端设备上的部署潜力。
- 3 未来需要结合自监督学习和轻量化设计，进一步增强模型的泛化能力和实用性。
Applications
Immediate Applications
智能手机摄影
提升手机拍照的清晰度和细节还原能力，改善低光环境下的图像质量，增强用户体验。
自动驾驶视觉系统
用于清晰还原道路和环境细节，提高车辆感知的准确性和安全性。
Long-term Vision
多模态场景重建
结合多源数据实现高质量三维场景重建，推动虚拟现实和增强现实的发展。
Abstract
Given a degraded input image, image restoration aims to recover the missing high-quality image content. Numerous applications demand effective image restoration, e.g., computational photography, surveillance, autonomous vehicles, and remote sensing. Significant advances in image restoration have been made in recent years, dominated by convolutional neural networks (CNNs). The widely-used CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatial details are preserved but the contextual information cannot be precisely encoded. In the latter case, generated outputs are semantically reliable but spatially less accurate. This paper presents a new architecture with a holistic goal of maintaining spatially-precise high-resolution representations through the entire network, and receiving complementary contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing the following key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) non-local attention mechanism for capturing contextual information, and (d) attention based multi-scale feature aggregation. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on six real image benchmark datasets demonstrate that our method, named as MIRNet-v2 , achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNetv2