Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Proposed Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST) achieves over 4dB PSNR improvement in spectral imaging reconstruction, combining MAP-based parameter estimation with a novel Transformer architecture.
Key Findings
Methodology
This paper introduces a MAP-based degradation-aware unfolding framework (DAUF) that estimates key parameters (α, β) from compressed measurements and physical masks via a dedicated estimator network (E). These parameters dynamically control each iteration, adjusting linear projections and denoising steps. The core innovation is the Half-Shuffle Transformer (HST), which employs a half-shuffle multi-head self-attention (HS-MSA) mechanism to jointly model local details and long-range dependencies efficiently. The multi-stage (9-stage) model outperforms existing state-of-the-art methods with significant PSNR gains (up to 38.36dB) while reducing computational costs, validated on simulated and real datasets.
Key Results
- DAUHST surpasses recent methods like BIRNAT and MST-L by over 4dB PSNR, achieving 38.36dB on average, with parameters at 28.15M and FLOPS at 19.42G, demonstrating superior efficiency and accuracy.
- In real-world experiments, DAUHST effectively reconstructs complex spectral data, with correlation coefficients reaching 0.9984, outperforming traditional and deep learning baselines visually and quantitatively.
- Ablation studies confirm the importance of parameter estimation and the half-shuffle mechanism, showing robustness across various hardware and noise conditions.
Significance
This work advances spectral compressive imaging by integrating physical degradation modeling into deep unfolding, leveraging Transformer architectures for high-dimensional data. It addresses longstanding challenges of capturing long-range dependencies and balancing computational efficiency, paving the way for real-time, high-fidelity spectral imaging in industrial and scientific applications.
Technical Contribution
The main technical contributions include a MAP-based parameter estimation module that adaptively controls the iterative process, and a novel half-shuffle Transformer (HST) that combines local and non-local attention with reduced complexity. This fusion of physical modeling and advanced neural architectures offers a new paradigm for high-dimensional image reconstruction, with theoretical and practical benefits over CNN-based methods.
Novelty
This is the first application of Transformer-based deep unfolding in spectral compressive imaging, with the half-shuffle mechanism enabling efficient long-range dependency modeling. It significantly differs from prior CNN or RNN approaches, offering a new direction for high-dimensional inverse problems.
Limitations
- Model performance may decline under extremely high noise or very high compression ratios, due to limitations in parameter estimation robustness.
- Current architecture demands substantial computational resources, posing challenges for deployment on embedded devices.
- Generalization to highly dynamic or unseen hardware configurations remains to be validated, requiring further adaptation.
Future Work
Future directions include enhancing parameter estimation robustness, developing lightweight Transformer variants, and extending the framework to adaptive real-time spectral imaging in diverse environments. Further research will focus on improving hardware compatibility and模型泛化能力，以实现更广泛的工业应用。
AI Executive Summary
Spectral imaging has become a vital technology for capturing detailed information across multiple wavelengths, with applications spanning remote sensing, medical diagnostics, and environmental monitoring. Traditional methods rely on spectral scanning, which is slow and unsuitable for dynamic scenes. Snapshot compressive imaging (SCI) systems like CASSI offer a promising alternative by capturing compressed measurements in a single shot, but reconstructing high-quality hyperspectral images remains a challenge due to the ill-posed inverse problem.
Existing solutions include model-based algorithms, plug-and-play methods, end-to-end deep networks, and deep unfolding techniques. While effective, these approaches face limitations: model-based methods require manual tuning, end-to-end models often ignore physical principles, and CNN-based deep unfolding struggles with long-range dependencies. Recently, Transformer architectures have shown remarkable success in modeling global interactions, but their high computational cost limits practicality.
This paper introduces a novel framework, DAUHST, that combines a MAP-based degradation-aware unfolding strategy with a half-shuffle Transformer denoiser. The parameter estimator network (E) learns to extract degradation patterns directly from measurements, guiding each iteration adaptively. The half-shuffle Transformer (HST) employs a half-shuffle multi-head self-attention mechanism to efficiently model both local details and long-range dependencies, significantly reducing computational complexity compared to traditional global Transformers.
Extensive experiments on simulated and real datasets demonstrate that DAUHST outperforms state-of-the-art methods by over 4dB PSNR, achieving a maximum of 38.36dB, with much lower computational costs. The model effectively reconstructs complex spectral information, maintaining high spectral fidelity and visual quality. Its robustness across different scenarios suggests strong potential for real-time high-dimensional imaging applications.
Looking ahead, the framework can be extended with more robust parameter estimation, lightweight Transformer variants, and broader hardware deployment. This work paves the way for high-fidelity, efficient spectral imaging, bridging physical modeling and deep learning innovations to meet the demands of future scientific and industrial needs.
Deep Dive
Abstract
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST