SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles
SafeBench integrates 8 safety-critical scenarios and 4 generation algorithms for comprehensive autonomous vehicle safety evaluation.
Key Findings
Methodology
Built on Carla, SafeBench features 8 critical scenarios, 4 generation algorithms (LC, AS, CS, AT), and evaluates 4 deep RL algorithms (PPO, SAC, DDPG, TD3) with 4 perception inputs. Large-scale simulations reveal that generated scenarios are more challenging, with collision rates exceeding 73%. The platform employs multi-metric evaluation including safety, functionality, and etiquette, enabling fair comparison across algorithms and scenarios. Scenario selection emphasizes high-risk, transferable cases, improving evaluation efficiency. The approach systematically analyzes performance trade-offs, transferability, and robustness, providing a comprehensive safety assessment framework.
Key Results
- Generated scenarios exhibit high challenge, with collision rates over 73%, validating their difficulty. LC and AT algorithms are most effective at inducing collisions, demonstrating the platform’s ability to generate risk-rich scenarios. Variability across scenarios and algorithms highlights the importance of diverse testing environments. Multi-metric results show significant performance differences, with some algorithms excelling in safety metrics while others perform better in functionality. The evaluation confirms the platform’s capacity to identify safety vulnerabilities and guide algorithm improvements.
- Analysis indicates certain scenarios, such as vehicle passing, are more difficult, with higher collision rates and transferability. Different generation algorithms produce varied effectiveness even within the same scenario. Metrics like red-light running, route completion, and deviation distances reveal nuanced performance profiles. The comprehensive evaluation underscores the necessity of multi-scenario, multi-metric testing to robustly validate autonomous driving systems. Overall, SafeBench achieves high efficiency and fairness, setting a new standard for safety benchmarking.
- The platform’s systematic, large-scale testing approach addresses current gaps in safety validation, enabling industry-wide adoption. By integrating diverse scenarios, generation methods, and evaluation metrics, it offers a scalable, extensible solution to accelerate safe deployment of autonomous vehicles. The findings support the development of more robust algorithms, fostering industry confidence and regulatory standards.
Significance
This work provides a unified, scalable framework for rigorous safety validation of autonomous driving systems, overcoming limitations of traditional real-world testing. By enabling large-scale, diverse, and fair evaluation, SafeBench accelerates industry adoption of safer algorithms, reduces safety risks, and promotes standardization. Its modular design facilitates future extensions, fostering collaborative research and industry-wide benchmarks. The platform’s comprehensive approach addresses critical gaps in safety assurance, supporting regulatory approval and public trust, ultimately contributing to safer autonomous transportation.
Technical Contribution
SafeBench introduces a modular, open-source platform integrating 8 safety-critical scenarios, 4 scenario generation algorithms, and multi-metric evaluation. It employs large-scale simulation with diverse perception inputs and deep RL algorithms, enabling detailed performance analysis. The platform’s scenario selection strategy enhances transferability and risk focus, while its multi-dimensional metrics provide a holistic safety assessment. This systematic integration and benchmarking approach significantly advances the state-of-the-art in autonomous vehicle safety evaluation, offering a scalable, extensible tool for academia and industry.
Novelty
This is the first comprehensive platform combining multi-scenario generation, multi-algorithm evaluation, and multi-metric assessment within a unified framework. It bridges the gap between isolated scenario testing and industry-standard benchmarks, enabling fair, large-scale comparison of autonomous driving algorithms. The integration of diverse generation algorithms (adversarial and knowledge-based) with deep RL-based controllers and multi-faceted metrics represents a significant innovation, setting a new standard for safety validation.
Limitations
- The platform relies on simulation data, which may not fully capture real-world complexities, limiting direct transferability. Some scenario generation algorithms depend heavily on parameter tuning, potentially introducing bias. Computational costs for large-scale testing remain high, constraining real-time or on-vehicle deployment. Future work should focus on integrating real-world data, optimizing algorithms for efficiency, and validating results through physical testing.
Future Work
Future directions include incorporating real traffic data to enhance scenario realism, developing adaptive scenario generation methods, and integrating hardware-in-the-loop testing. Expanding the scenario library to cover more edge cases and rare events will improve robustness. Additionally, efforts will focus on reducing computational costs and enabling real-time evaluation, facilitating deployment on actual vehicles. Strengthening industry collaborations and establishing standardized benchmarks will further promote safe autonomous driving development.
AI Executive Summary
As autonomous driving (AD) technology advances rapidly, ensuring safety remains a critical challenge. Traditional testing relies heavily on real-world driving, requiring hundreds of millions of miles to encounter rare safety-critical scenarios, which is costly and inefficient. To address this, the authors introduce SafeBench, a comprehensive simulation-based benchmarking platform built on Carla, designed to evaluate AD algorithms across a wide spectrum of safety-critical scenarios.
SafeBench integrates 8 representative scenarios, such as lane changing and crossing negotiation, each with 10 diverse route variants. It employs four scenario generation algorithms—LC, AS, CS, and AT—to produce challenging, transferable scenarios that induce safety violations. The platform evaluates four deep reinforcement learning-based AD agents (PPO, SAC, DDPG, TD3) with four perception input types, ensuring fairness and diversity in assessment.
Large-scale simulations reveal that generated scenarios are significantly more challenging than naturalistic data, with collision rates exceeding 73% in some cases. The platform’s multi-metric evaluation system captures safety, functionality, and etiquette, providing a holistic performance picture. Results demonstrate notable trade-offs between safety and efficiency, emphasizing the importance of diverse testing environments.
This unified framework addresses key industry gaps, offering a scalable, extensible tool for safety validation. It facilitates algorithm development, regulatory standards, and industry benchmarking, ultimately accelerating the deployment of safer autonomous vehicles. Future work aims to incorporate real-world data, optimize computational efficiency, and expand scenario diversity, paving the way for industry-wide safety assurance.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家大型厨房里准备一道复杂的菜肴。每次做菜都需要不同的原料和步骤，有时会遇到突发状况，比如火太大或食材不新鲜。为了确保菜肴安全、味道好，你需要模拟各种可能出现的问题，提前练习应对方法。SafeBench就像这个厨房的智能助手，它帮你设计各种“突发事件”场景，比如突然出现的油烟、食材短缺或火灾，然后测试你的厨艺（自动驾驶算法）是否能应对。这些场景多样、难度不同，帮助你不断改进厨艺，确保菜肴安全、完美。通过这种模拟训练，你可以提前发现潜在风险，避免在真正的厨房里出错。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的赛车游戏。你要在不同的赛道上跑，有时候会遇到突然出现的障碍物，比如路上突然出现一只动物或者一个车突然变道。这时候，你的车（自动驾驶系统）需要快速反应，避免撞到东西。为了让你的车变得更聪明，游戏开发者设计了各种不同的“挑战场景”，让车在模拟环境中练习应对各种突发情况。这些场景有的很难，有的比较简单。开发者还用不同的“训练方法”来测试车的反应速度和安全性，比如模拟敌人车辆的恶意行为，或者用规则引导场景。最后，他们会用一些评分标准，比如撞车次数、是否闯红灯、能否按时完成路线，来评价车的表现。这样一来，车在真实道路上遇到危险时，就能更聪明、更安全。这就像你在游戏中不断练习，变得越来越厉害，最后在真实比赛中也能表现出色。
Abstract
As shown by recent studies, machine intelligence-enabled systems are vulnerable to test cases resulting from either adversarial manipulation or natural distribution shifts. This has raised great concerns about deploying machine learning algorithms for real-world applications, especially in safety-critical domains such as autonomous driving (AD). On the other hand, traditional AD testing on naturalistic scenarios requires hundreds of millions of driving miles due to the high dimensionality and rareness of the safety-critical scenarios in the real world. As a result, several approaches for autonomous driving evaluation have been explored, which are usually, however, based on different simulation platforms, types of safety-critical scenarios, scenario generation algorithms, and driving route variations. Thus, despite a large amount of effort in autonomous driving testing, it is still challenging to compare and understand the effectiveness and efficiency of different testing scenario generation algorithms and testing mechanisms under similar conditions. In this paper, we aim to provide the first unified platform SafeBench to integrate different types of safety-critical testing scenarios, scenario generation algorithms, and other variations such as driving routes and environments. Meanwhile, we implement 4 deep reinforcement learning-based AD algorithms with 4 types of input (e.g., bird's-eye view, camera) to perform fair comparisons on SafeBench. We find our generated testing scenarios are indeed more challenging and observe the trade-off between the performance of AD agents under benign and safety-critical testing scenarios. We believe our unified platform SafeBench for large-scale and effective autonomous driving testing will motivate the development of new testing scenario generation and safe AD algorithms. SafeBench is available at https://safebench.github.io.