Bilevel Layer-Positioning LoRA for Real Image Dehazing
Proposed BiLaLoRA with CLIP-based H2C loss achieves superior real-world image dehazing with efficient parameter tuning.
Key Findings
Methodology
This paper introduces the H2C (Haze-to-Clear) loss leveraging CLIP's cross-modal capabilities, reformulating dehazing as a semantic alignment in latent space. The BiLaLoRA strategy employs bilevel optimization to automatically search for critical network layers for injection, combining low-rank adaptation for parameter efficiency. The process involves extracting semantic features from haze and clear images via CLIP, defining the H2C loss to align semantic directions, and using a bi-level framework to optimize both layer positions and LoRA parameters. Extensive experiments on RTTS, URHI, and Fattal datasets demonstrate the method's superior performance in real-world scenarios, especially under complex lighting and dense haze conditions.
Key Results
- On RTTS, URHI, and Fattal datasets, BiLaLoRA outperforms SOTA methods with an average improvement of 4.5% across FADE, BIQME, Entropy, and MUSIQ metrics, especially excelling in nighttime and dense haze scenes.
- The H2C loss enhances cross-domain generalization, enabling adaptation to diverse haze patterns without paired data. Fine-tuning time is reduced to about 0.94 hours, with parameters comparable to full fine-tuning, demonstrating efficiency.
- Ablation studies confirm the importance of the directional semantic loss and automatic layer search, with the model maintaining robustness across multiple domains and scenarios.
Significance
This work addresses key challenges in real-world image dehazing: lack of supervised data, high computational cost of full fine-tuning, and poor generalization. By integrating cross-modal semantic guidance and parameter-efficient adaptation, it offers a scalable solution for deploying dehazing models in practical applications such as autonomous driving and surveillance. The approach paves the way for leveraging vision-language models to enhance low-level vision tasks, reducing reliance on labeled datasets and enabling rapid adaptation to new environments.
Technical Contribution
The core innovation lies in the H2C loss, which aligns semantic transformations in latent space guided by text prompts, enabling unsupervised learning. The BiLaLoRA framework introduces a bilevel optimization process to automatically identify the most impactful network layers for injection, combining low-rank adaptation with architecture search. This results in a parameter-efficient, flexible, and scalable method that significantly reduces training time while maintaining or surpassing state-of-the-art performance. The method's generality is validated across multiple architectures and datasets.
Novelty
This is the first work to incorporate CLIP's cross-modal semantic capabilities into real image dehazing, transforming the task into a latent space alignment problem guided by text prompts. The automatic layer search via bilevel optimization for LoRA modules is a novel engineering solution, enabling dynamic adaptation without manual layer selection. Unlike prior domain adaptation or supervised methods, this approach achieves high performance with minimal parameter updates, representing a significant step forward in unsupervised low-level vision tasks.
Limitations
- The reliance on pre-trained CLIP limits performance in scenarios where semantic representations are weak or ambiguous, such as extreme weather or unusual scenes.
- The bilevel optimization, while efficient, may still face scalability issues with very large models or multiple simultaneous tasks, requiring further algorithmic refinement.
- Robustness under extreme lighting or weather conditions needs further validation, and future work could incorporate multi-modal cues for enhanced stability.
Future Work
Future directions include integrating multi-modal data (e.g., depth, thermal) to improve robustness, developing more scalable bilevel optimization algorithms, and extending the framework to other low-level vision tasks like super-resolution and image restoration. Additionally, exploring self-supervised training paradigms could reduce dependence on pre-trained models, broadening applicability.
AI Executive Summary
Image dehazing remains a critical challenge in low-level vision, especially in real-world scenarios characterized by complex lighting, dense fog, and diverse weather conditions. Traditional methods relied heavily on handcrafted priors, which often failed under real scene complexities. Deep learning approaches, while powerful, struggled with domain gaps and high computational costs when adapting to new environments. This paper introduces a novel framework that leverages the cross-modal capabilities of CLIP to reformulate dehazing as a semantic alignment problem in latent space, bypassing the need for paired supervision.
The core innovation is the H2C (Haze-to-Clear) loss, which guides the model to align the semantic transformation of hazy images toward clear images based on text prompts. This enables effective unsupervised learning across diverse scenes. To address the high cost of full model fine-tuning, the authors propose BiLaLoRA, a bilevel optimization-based parameter-efficient strategy that automatically searches for the most impactful network layers for injection, combining low-rank adaptation with architecture search. This approach significantly reduces training time while maintaining or improving performance.
Extensive experiments on datasets like RTTS, URHI, and Fattal demonstrate that the proposed method outperforms state-of-the-art techniques, especially in nighttime and dense haze conditions. The model generalizes well across different domains and scene types, with minimal computational overhead. The results highlight the potential for deploying lightweight, adaptable dehazing models in real-time applications such as autonomous driving, surveillance, and augmented reality.
While promising, the approach still faces limitations related to the dependency on pre-trained CLIP's semantic understanding in extreme scenarios and the scalability of bilevel optimization. Future work aims to incorporate multi-modal cues, enhance robustness, and extend the framework to other low-level vision tasks, fostering broader impact in intelligent visual systems.
Deep Dive
Abstract
Learning-based real image dehazing methods have achieved notable progress, yet they still face adaptation challenges in diverse real haze scenes. These challenges mainly stem from the lack of effective unsupervised mechanisms for unlabeled data and the heavy cost of full model fine-tuning. To address these challenges, we propose the haze-to-clear text-directed loss that leverages CLIP's cross-modal capabilities to reformulate real image dehazing as a semantic alignment problem in latent space, thereby providing explicit unsupervised cross-modal guidance in the absence of reference images. Furthermore, we introduce the Bilevel Layer-positioning LoRA (BiLaLoRA) strategy, which learns both the LoRA parameters and automatically search the injection layers, enabling targeted adaptation of critical network layers. Extensive experiments demonstrate our superiority against state-of-the-art methods on multiple real-world dehazing benchmarks. The code is publicly available at https://github.com/YanZhang-zy/BiLaLoRA.