Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

Recent advances in vision-language models (VLMs), exemplified by architectures like CLIP, BLIP, and Florence, have significantly improved multimodal scene understanding. These models can perform complex tasks such as spatial reasoning, object detection, and natural language question answering, which are crucial for autonomous driving. Meanwhile, the evolution of autonomous vehicles from rule-based systems to deep learning end-to-end architectures has emphasized perception, reasoning, and control integration. Existing benchmarks like BDD-X, DriveBench, and NuPrompt primarily focus on static perception and single-turn reasoning, lacking the capacity to evaluate continuous, interactive decision-making. Closed-loop evaluation methods, such as DriveLM and Bench2Drive, have begun to address this gap but are limited to trajectory or maneuver correctness, without assessing multimodal reasoning quality or dynamic scene understanding. The need for a comprehensive, real-time, multimodal evaluation framework remains urgent to bridge the gap between perception and decision-making in complex environments.

Current evaluation approaches for VLM-based autonomous driving predominantly rely on static datasets and open-loop metrics, which cannot reflect the model’s performance during continuous, real-time interactions. Static datasets lack the temporal and causal context necessary to assess decision consistency and robustness, especially in out-of-distribution or extreme scenarios like off-road deviations. Existing closed-loop systems mainly focus on trajectory accuracy, ignoring the model’s reasoning process and multimodal understanding. This disconnect hampers the development of models capable of reasoning about complex scenes, explaining their decisions, and recovering from errors. The core challenge lies in generating high-quality, behavior-grounded question-answer pairs dynamically during simulation, which requires sophisticated scene understanding and causal reasoning. Overcoming this bottleneck is essential for advancing autonomous systems that are safe, reliable, and capable of handling unpredictable real-world conditions.

The key innovations include: 1) DriveCommenter, an automated, scene-aware QA generator that produces behavior-grounded questions covering perception, prediction, and decision-making in real-time; 2) a unified multimodal reasoning framework supporting multi-format visual inputs and graph-based chain-of-thought inference; 3) seamless integration of VLMs into CARLA for end-to-end closed-loop evaluation, enabling continuous perception, reasoning, and control; 4) a comprehensive ecosystem with visualization, debugging, and large-scale annotation tools to facilitate research and benchmarking. These advancements enable the evaluation of VLMs’ full capabilities in dynamic, out-of-distribution scenarios, addressing the limitations of static datasets and trajectory-only metrics, and fostering the development of more robust autonomous driving models.

• �� DriveCommenter extracts scene information from CARLA, including actor positions, traffic signals, weather, and map topology, to generate diverse, behavior-grounded QA pairs. • The VLM receives sensor data (images, BEV maps, textual commands), performs multimodal perception, and employs a graph-of-thought architecture to execute multi-step reasoning. • Based on reasoning outputs, the model generates natural language control commands, which are converted into vehicle control signals via an action module. • A VQA evaluator compares the model’s answers with DriveCommenter’s ground truth, scoring perception and reasoning accuracy. • The entire pipeline supports real-time operation, with data stored for offline analysis and model refinement. • Visualization and debugging tools assist in data inspection, error analysis, and scenario editing, ensuring system robustness.

Experiments involve evaluating VLM agents across diverse scenarios in CARLA, including normal, off-route, and off-road deviations. Baselines include recent multimodal models like BLIP2 and Florence, combined with chain-of-thought reasoning modules. Metrics include QA accuracy, control command correctness, and safety-related deviations. Hyperparameters such as reasoning depth, sensor resolution, and scene complexity are tuned systematically. Ablation studies assess the impact of DriveCommenter-generated QA quality and reasoning architecture on overall performance. The evaluation emphasizes robustness in out-of-distribution scenarios, with repeated trials to ensure statistical significance. Results demonstrate that the proposed framework significantly improves decision accuracy and safety metrics compared to traditional open-loop benchmarks.

The system achieved 85% accuracy in behavior-grounded QA in complex scenarios, outperforming static benchmarks by 15%. Chain-of-thought reasoning reduced cumulative decision errors by 20%, and the model maintained stable performance across 1000 diverse scenarios. The approach improved safety metrics, including collision avoidance and lane keeping, by 10%. Ablation experiments confirmed that DriveCommenter’s QA quality directly correlates with model robustness, emphasizing the importance of dynamic, behavior-aware evaluation. These results validate the effectiveness of the integrated framework in realistic, out-of-distribution driving conditions.

This framework can be employed for comprehensive testing of autonomous driving algorithms, enabling developers to evaluate perception, reasoning, and control in a unified environment. It supports industry R&D, simulation-based validation, and safety certification, accelerating deployment of reliable autonomous systems. Additionally, it can serve as a training platform for reinforcement learning agents, providing rich, behavior-grounded feedback to improve decision-making policies. Long-term, the system could be integrated into real vehicles for continuous learning and adaptation, fostering safer and more intelligent autonomous driving solutions.

The current system relies heavily on high-quality scene annotations and simulation fidelity, which may limit scalability and transferability to real-world scenarios. Its performance under adverse weather, nighttime, or highly cluttered environments remains limited. The computational cost of multimodal perception and reasoning is significant, posing challenges for real-time deployment on resource-constrained platforms. Furthermore, the generalization to highly complex or novel scenarios needs further validation, and future work should focus on robustness enhancement, multi-sensor fusion, and real-world testing to address these issues.