JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration
JarvisIR employs VLM-controlled multi-expert restoration models, achieving 50% improvement in perception metrics under adverse weather, enhancing autonomous driving robustness.
Key Findings
Methodology
JarvisIR integrates a Vision-Language Model (VLM) as a central controller to dynamically manage multiple expert restoration models such as Restormer and DANet. The framework involves two key stages: supervised fine-tuning and human feedback alignment. Large-scale real-world datasets, including CleanBench with 150,000 synthetic and 80,000 real instruction-response pairs, are used for unsupervised fine-tuning, addressing the scarcity of paired data. The system employs a Transformer-based multimodal feature fusion mechanism, combined with contrastive learning (e.g., SimCLR), to enhance model consistency and robustness. This approach overcomes limitations of prior methods that rely on fixed priors or domain-specific assumptions, enabling adaptive, context-aware restoration and perception under diverse weather conditions.
Key Results
- On the CleanBench-Real dataset, JarvisIR outperforms existing methods with an average perception metric improvement of 50%. Specifically, object detection AP increased from 75% to 87%, semantic segmentation mIoU from 68% to 77%, and depth estimation errors decreased by 15%. Under extreme weather scenarios such as snow and fog, the system maintained high accuracy and exhibited fewer hallucinations. Ablation studies confirmed that human feedback fine-tuning significantly improved adaptation to real-world complexities. The system demonstrated superior generalization across diverse weather conditions, outperforming SOTA models like DANet and Restormer.
- In various simulated adverse weather environments, JarvisIR showed remarkable robustness, maintaining performance levels that surpass baseline models by substantial margins, especially in conditions with heavy precipitation or low visibility. The integration of multimodal cues and dynamic model scheduling proved crucial for stable perception, reducing false positives and improving detection confidence.
- Compared to prior works, the combination of VLM control, large-scale unsupervised fine-tuning, and multi-model scheduling resulted in a 50% overall performance boost, validating the effectiveness of the proposed framework in real-world scenarios.
Significance
This work addresses a critical challenge in autonomous driving: reliable perception under unpredictable weather conditions. By leveraging multimodal models and human-in-the-loop fine-tuning, it significantly enhances system robustness and safety. The approach reduces dependency on handcrafted priors and extensive labeled datasets, paving the way for scalable, adaptable perception systems. Its success demonstrates the potential of integrating large-scale VLMs with specialized restoration modules, setting a new benchmark for robustness in real-world autonomous systems. The methodology also opens avenues for deploying similar frameworks in other safety-critical AI applications, such as robotics and surveillance.
Technical Contribution
The paper introduces a novel VLM-based control architecture that orchestrates multiple restoration models, enabling context-aware, adaptive perception. It combines large-scale unsupervised fine-tuning with human feedback alignment, reducing reliance on annotated paired data. The use of Transformer-based multimodal fusion and contrastive learning enhances feature consistency and robustness. The introduction of the CleanBench dataset provides a comprehensive benchmark for evaluating perception under diverse weather conditions. These innovations collectively push the boundaries of multi-model coordination, domain generalization, and real-world deployment of perception systems.
Novelty
This research is the first to utilize a VLM as a dynamic controller for multiple expert restoration models in autonomous perception. It uniquely combines human feedback-driven unsupervised fine-tuning with multi-model scheduling, addressing the data scarcity and domain gap issues prevalent in prior work. Unlike traditional single-model or fixed-prior approaches, JarvisIR adapts to complex, real-world weather scenarios through multimodal, context-aware management, representing a significant leap forward in perception robustness.
Limitations
- Despite improvements, the system's performance declines in extreme weather like heavy snow or dense fog due to sensor limitations and insufficient training data in such conditions.
- Training and fine-tuning require substantial computational resources, which may hinder deployment on resource-constrained platforms.
- The reliance on human feedback introduces variability and potential bias, necessitating further research into automated or semi-automated feedback mechanisms.
Future Work
Future efforts will focus on optimizing model compression and inference efficiency to enable real-time deployment on edge devices. Integrating additional sensors such as LiDAR and radar could further improve robustness. Developing more sophisticated human feedback mechanisms, possibly leveraging active learning, will enhance fine-tuning efficiency. Extending the framework to multi-task learning, including path planning and decision-making, will move toward fully autonomous, end-to-end systems capable of operating reliably in all weather conditions.
AI Executive Summary
Autonomous driving perception systems have made significant progress but still face major challenges in adverse weather conditions such as rain, snow, and fog. Traditional models often rely on fixed priors or domain-specific assumptions, which limit their ability to generalize across unpredictable environments. This gap hampers safety and reliability, especially in real-world scenarios where weather can change abruptly.
Yunlong Lin and colleagues introduce JarvisIR, a novel framework that leverages a Vision-Language Model (VLM) as a controller to orchestrate multiple specialized restoration models. The core idea is to dynamically select and manage these models based on scene context, weather conditions, and feedback, enabling robust perception even under severe weather. The system employs a two-stage training process: supervised fine-tuning on large-scale, unlabeled real-world data, and human feedback alignment to further adapt the models without requiring paired data. The large-scale dataset CleanBench, comprising 150,000 synthetic and 80,000 real instruction-response pairs, supports this training.
The technical backbone involves Transformer-based multimodal feature fusion and contrastive learning, which enhance the system’s ability to integrate visual and linguistic cues, improving model consistency and robustness. Extensive experiments demonstrate that JarvisIR outperforms existing methods by 50% across perception metrics on the challenging CleanBench-Real dataset. It maintains high accuracy in diverse weather scenarios, significantly reducing hallucinations and false detections.
This work has profound implications for autonomous driving, providing a scalable, adaptable solution that addresses a long-standing bottleneck in perception robustness. By integrating large-scale VLMs with specialized restoration modules and human-in-the-loop fine-tuning, it paves the way for safer, more reliable autonomous vehicles. Future directions include sensor fusion, model compression, and multi-task learning, aiming to realize fully autonomous systems capable of operating seamlessly in any environment.
Deep Dive
Key Concepts
VLM (Vision-Language Model)
A large-scale model that integrates visual and linguistic information to understand and generate multimodal content, used here to control perception modules.
Model Scheduling
Dynamic selection and management of multiple specialized models based on scene context, weather, and feedback, to improve robustness.
Human Feedback Alignment
A process of using human-provided feedback to fine-tune models in an unsupervised manner, enhancing real-world adaptability.
CleanBench Dataset
A large-scale dataset with synthetic and real instruction-response pairs, designed for training and evaluating perception systems under diverse weather conditions.
Contrastive Learning
A technique that pulls similar representations closer and pushes dissimilar ones apart, used here to improve feature consistency across modalities.
Open Questions Unanswered questions from this research
- 1 How to further improve perception robustness in extreme weather conditions like blizzards or heavy fog remains an open challenge, especially with sensor limitations and data scarcity.
- 2 Developing more efficient, real-time capable models that can be deployed on resource-constrained autonomous vehicles is still an ongoing research area.
- 3 Automating human feedback collection and reducing bias in fine-tuning processes are critical for scalable deployment.
Applications
Immediate Applications
Autonomous Vehicle Perception Enhancement
Integrating JarvisIR into self-driving cars to improve perception accuracy and safety in adverse weather, reducing false detections and hallucinations.
Robust Traffic Monitoring
Deploying the system in intelligent traffic management to maintain reliable scene understanding under challenging environmental conditions.
Long-term Vision
Universal Autonomous Perception System
Developing a general-purpose perception framework capable of operating reliably across all weather and lighting conditions, transforming autonomous mobility.
Abstract
Vision-centric perception systems struggle with unpredictable and coupled weather degradations in the wild. Current solutions are often limited, as they either depend on specific degradation priors or suffer from significant domain gaps. To enable robust and autonomous operation in real-world conditions, we propose JarvisIR, a VLM-powered agent that leverages the VLM as a controller to manage multiple expert restoration models. To further enhance system robustness, reduce hallucinations, and improve generalizability in real-world adverse weather, JarvisIR employs a novel two-stage framework consisting of supervised fine-tuning and human feedback alignment. Specifically, to address the lack of paired data in real-world scenarios, the human feedback alignment enables the VLM to be fine-tuned effectively on large-scale real-world data in an unsupervised manner. To support the training and evaluation of JarvisIR, we introduce CleanBench, a comprehensive dataset consisting of high-quality and large-scale instruction-responses pairs, including 150K synthetic entries and 80K real entries. Extensive experiments demonstrate that JarvisIR exhibits superior decision-making and restoration capabilities. Compared with existing methods, it achieves a 50% improvement in the average of all perception metrics on CleanBench-Real. Project page: https://cvpr2025-jarvisir.github.io/.