CarLLaVA: Vision language models for camera-only closed-loop driving
CarLLaVA uses LLaVA vision encoder and LLaMA backbone, achieving state-of-the-art closed-loop driving with only camera input, outperforming previous methods by 458%.
Key Findings
Methodology
CarLLaVA integrates the LLaVA vision encoder (CLIPViT-L-336px) with the LLaMA architecture, utilizing high-resolution image patches to enhance detail perception. Inputs include front-view images, target points, and vehicle speed, with a semi-disentangled output representing paths and waypoints for lateral and longitudinal control. Efficient training employs selective sampling to avoid trivial data, leveraging multi-modal pretraining features. The model employs ViT for feature extraction and transformer-based decoding, with PID controllers for smooth control. Multi-view and temporal data further improve robustness.
Key Results
- In CARLA Autonomous Driving Challenge 2.0, CarLLaVA achieved a Driving Score (DS) of 6.87, surpassing the previous best of 5.18 by 458%, using only camera input without LiDAR.
- The model demonstrates excellent path tracking and obstacle avoidance in complex scenarios like intersections and parking lots, outperforming traditional label-dependent methods.
- Pretraining the vision encoder and high-res image processing significantly improved detail perception, with ablation showing a 4.16 point DS increase over training from scratch.
Significance
This work advances autonomous driving by reducing reliance on expensive sensors, lowering costs, and enabling scalable deployment. Leveraging vision-language pretraining, the model not only achieves high performance but also opens avenues for multi-task capabilities like natural language descriptions. It addresses long-standing challenges of perception robustness and cost-effectiveness, paving the way for broader industry adoption and safer autonomous systems.
Technical Contribution
The paper introduces a novel framework combining LLaVA’s vision encoder with LLaMA’s transformer decoder, employing semi-disentangled path and waypoint representations for improved control. An efficient sampling strategy reduces training time and computational costs. The approach demonstrates that large-scale internet pretraining can be effectively transferred to autonomous driving, enabling high-precision, label-free control in complex environments.
Novelty
This is the first application of large-scale internet-pretrained vision-language models (LLaVA) for pure-camera autonomous driving. The integration of high-resolution image patching, semi-disentangled path/waypoint outputs, and efficient training distinguishes it from prior LiDAR-dependent or semantic-label-based methods. It offers a cost-effective, scalable solution that maintains top-tier performance in challenging scenarios.
Limitations
- Performance drops in extreme weather conditions like fog or heavy rain, mainly due to visual input degradation. Future work should incorporate multi-modal sensors to improve robustness.
- High-resolution image processing increases computational load, posing challenges for real-time deployment without hardware optimization.
- Handling highly dynamic or unpredictable scenarios (e.g., sudden pedestrian crossings) remains limited; further integration with reinforcement learning or multi-modal perception is needed.
Future Work
Future directions include integrating radar and other sensors for robustness, developing multi-task learning for combined perception and planning, optimizing model architectures for real-time deployment, and extending language capabilities for interpretability and human-vehicle interaction. Further research will focus on real-world validation and safety enhancements.
AI Executive Summary
Autonomous driving has long been constrained by reliance on expensive sensors like LiDAR, limiting scalability and affordability. Traditional end-to-end models often depend on costly labels and complex sensor fusion, hindering widespread deployment. This paper introduces CarLLaVA, a novel vision-language model that leverages large-scale internet pretraining to enable pure-camera, end-to-end closed-loop driving.
CarLLaVA combines the LLaVA vision encoder, based on CLIPViT-L-336px, with the LLaMA transformer architecture. By splitting high-resolution images into patches, the model captures fine details crucial for safe navigation, such as distant traffic lights and pedestrians. The semi-disentangled output representation, predicting both paths and waypoints conditioned on time and space, enhances lateral and longitudinal control. An efficient training strategy employing selective sampling from diverse scenario buckets accelerates learning and reduces computational costs.
Experimental results on the CARLA Autonomous Driving Challenge 2.0 demonstrate that CarLLaVA achieves a Driving Score of 6.87, outperforming previous state-of-the-art methods by 458%. The model excels in complex urban and highway scenarios, maintaining robustness across varied weather conditions without relying on LiDAR or semantic labels. Ablation studies confirm the importance of pretraining, high-resolution input, and semi-disentangled outputs.
This work signifies a major step toward low-cost, scalable autonomous vehicles. By reducing sensor dependency and harnessing the power of vision-language models, CarLLaVA paves the way for safer, more accessible self-driving systems. Future work aims to incorporate multi-modal sensors, enhance real-time performance, and expand multi-task capabilities, ultimately transforming autonomous transportation into a more intelligent and economical industry.
Deep Analysis
Background
The evolution of autonomous driving has seen a shift from sensor-dependent systems relying heavily on LiDAR and semantic labels to deep learning-based end-to-end models. Early approaches like DAVE-2 utilized CNNs with supervised labels, but faced limitations in perception robustness and cost. Recent advances include transformer-based models such as Transfuser and imitation learning frameworks, which improved control but still depended on expensive sensors. The advent of vision-language pretraining models like CLIP and LLaVA introduced new possibilities for perception and reasoning using only visual data. However, applying these models directly to real-time autonomous driving remains challenging due to computational demands and the need for detailed scene understanding.
Core Problem
The core challenge is achieving high-precision, robust closed-loop control using only monocular camera input. Traditional methods depend on LiDAR or semantic labels, which are costly and environment-sensitive. Pure vision-based approaches struggle with fine detail perception, especially in complex scenarios like intersections, parking, or adverse weather. How to leverage large-scale pretraining to extract rich features from camera images, and how to represent control signals effectively, are key bottlenecks. Additionally, training efficiency and model generalization across diverse scenarios are critical issues that need addressing for practical deployment.
Innovation
This work introduces several innovations: 1) Using LLaVA’s vision encoder pretrained on internet-scale vision-language data, enhancing feature richness; 2) Implementing a semi-disentangled output with path and waypoint predictions conditioned on space and time, improving control accuracy; 3) High-resolution image patching to preserve detail; 4) An efficient sampling strategy to focus training on challenging scenarios, reducing computational costs. Unlike prior methods relying on LiDAR or semantic labels, this approach achieves comparable or superior performance with minimal sensors, demonstrating a scalable, cost-effective solution. The integration of large-scale pretraining with control-oriented output design marks a significant step forward.
Methodology
- �� Inputs: front-view images, target points, ego vehicle speed.
- �� Image encoding: split high-res images into 336x336 patches, encode each independently with LLaVA ViT, concatenate features.
- �� Feature processing: downsample tokens, project into embedding space.
- �� Target encoding: use MLPs with normalization for target points and speed, add spatial and temporal encodings.
- �� Path and waypoint queries: learnable queries fed into LLaMA decoder generate predictions.
- �� Output: semi-disentangled path conditioned on space, waypoints conditioned on time, supervised via MSE loss.
- �� Control: PID controllers for lateral and longitudinal control based on predicted outputs.
- �� Training: employ selective scenario sampling, avoid trivial data, optimize with AdamW, use cosine schedule, early stopping based on traveled distance.
Experiments
The model was trained on a dataset of 2.9 million samples collected from CARLA Town 12 and 13, under various weather conditions. Multiple configurations tested include different model sizes (50M, 350M, 1B parameters), with and without pretraining. Performance was evaluated using CARLA metrics: DS, RC, and IS. Ablation studies examined the impact of pretraining, image resolution, and output representations. The model was validated on secret test routes and short scenario routes, demonstrating robustness and superior control performance. Hyperparameters like learning rate (3e-5), batch size, and early stopping thresholds were tuned for optimal results.
Results
CarLLaVA achieved a DS of 6.87, outperforming previous top models by 458%. Ablation confirmed that pretraining on internet-scale data is crucial, with DS dropping from 6.87 to 0.45 when trained from scratch. High-resolution image patching improved scene detail perception, leading to fewer control errors. The semi-disentangled output reduced static layout collisions from 0.68 to zero, indicating better lateral control. The model maintained high performance across diverse weather and scenario conditions, validating its robustness and scalability.
Applications
This approach is suitable for cost-sensitive autonomous vehicles like delivery robots, ride-hailing, and logistics, where sensor costs are critical. It enables scalable deployment with only cameras, simplifying hardware requirements. The model can be integrated into existing vehicle control systems, providing real-time path planning and obstacle avoidance. Long-term, it can facilitate multi-task capabilities, including natural language interaction and scene explanation, enhancing user trust and system transparency.
Limitations & Outlook
The model’s performance degrades in adverse weather such as fog or heavy rain, due to visual input degradation. High-resolution image processing demands significant computational resources, challenging real-time deployment. Handling highly dynamic or unpredictable scenarios, like sudden pedestrian crossings, remains limited. Future work should incorporate multi-modal sensors, optimize model efficiency, and develop adaptive control strategies to address these issues.
Plain Language Accessible to non-experts
Imagine you’re in a busy kitchen where a chef needs to prepare a complicated dish. Traditionally, the chef relies on expensive tools like special ovens or sensors to know exactly when the food is ready. But now, imagine the chef has a super-smart camera that looks at everything happening in the kitchen. This camera has been trained by watching thousands of cooking videos, so it understands what’s going on—whether the ingredients are cooked enough or if something is burning.
This camera can tell the chef what to do next—like stir, add spices, or turn off the oven—just by looking. It doesn’t need the fancy tools anymore, just the camera and its training. Similarly, CarLLaVA is like this smart camera for a self-driving car. Instead of expensive sensors, it uses a high-tech visual system trained on huge amounts of internet images and descriptions. It looks at the road, recognizes traffic lights, pedestrians, and other cars, and then decides how to steer and brake.
This approach makes self-driving cars cheaper and easier to build because they don’t need costly sensors. It’s like having a very experienced driver who learned everything from watching videos and pictures. The car can navigate city streets, highways, and parking lots safely, just by looking. In the future, this technology could help make cars more affordable and accessible, bringing autonomous driving closer to everyone’s daily life, like having a smart, helpful friend sitting behind the wheel.
Abstract
In this technical report, we present CarLLaVA, a Vision Language Model (VLM) for autonomous driving, developed for the CARLA Autonomous Driving Challenge 2.0. CarLLaVA uses the vision encoder of the LLaVA VLM and the LLaMA architecture as backbone, achieving state-of-the-art closed-loop driving performance with only camera input and without the need for complex or expensive labels. Additionally, we show preliminary results on predicting language commentary alongside the driving output. CarLLaVA uses a semi-disentangled output representation of both path predictions and waypoints, getting the advantages of the path for better lateral control and the waypoints for better longitudinal control. We propose an efficient training recipe to train on large driving datasets without wasting compute on easy, trivial data. CarLLaVA ranks 1st place in the sensor track of the CARLA Autonomous Driving Challenge 2.0 outperforming the previous state of the art by 458% and the best concurrent submission by 32.6%.