Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective
Introduced LanEvil++ benchmark, evaluating lane perception robustness; models drop 5.27% accuracy under environmental illusions.
Key Findings
Methodology
LanEvil++ benchmark uses CARLA to generate 94 high-fidelity scenes with 14 illusion types. By designing multi-level perturbations—shadows, reflections, road damages—the study systematically assesses lane detection (LD) and vision-language models (ADVLMs). Metrics include accuracy, F1-score, GPT-score, combined with closed-loop simulations and real-world case studies. The authors introduce MIDA, a hard-sample augmentation method, to improve model robustness against illusions.
Key Results
- Models experienced an average accuracy drop of 5.27%, with F1-score decreasing by 10.49%. Shadows caused the largest accuracy decline of 7.20%. ADVLMs' GPT-score decreased by 2.03%, with traffic obstruction illusions causing a 3.51% drop. MIDA improved robustness by 4.23% for LD and 3.82% for ADVLMs.
- Closed-loop simulations revealed that illusions could lead to incorrect driving decisions, highlighting safety risks. Real-world case studies confirmed these vulnerabilities, emphasizing the need for robustness enhancement.
- The benchmark and defense strategies provide a comprehensive framework for evaluating and improving perception models under environmental disturbances.
Significance
This work addresses a critical gap by systematically evaluating how natural illusions like shadows and reflections impair autonomous driving perception. It offers a standardized benchmark, facilitating industry-wide safety assessments. The findings underscore the importance of robustness in real-world deployment, aiming to reduce accident risks caused by environmental factors. The introduced MIDA method demonstrates practical potential for enhancing model resilience, contributing to safer autonomous systems. Overall, this research advances both theoretical understanding and practical solutions for environmental robustness in autonomous driving.
Technical Contribution
The paper pioneers the LanEvil++ benchmark, covering 14 illusion types with large-scale annotated data. It innovates by integrating multi-level perturbations in simulation, enabling detailed robustness evaluation. The MIDA approach employs hard sample mining and attention mechanisms, significantly boosting model resistance. These contributions differ from prior work by providing a systematic, multi-dimensional assessment platform and targeted defense strategies, pushing the frontier of perception robustness research.
Novelty
First comprehensive benchmark covering diverse environmental illusions affecting lane perception, combining simulation and real-world validation. The multi-level perturbation design and MIDA defense are novel, offering a new paradigm for robustness evaluation and enhancement. Unlike previous studies limited to specific scenarios or single models, this work provides a broad, systematic framework applicable across multiple perception architectures, setting a new standard in the field.
Limitations
- Simulation scenarios, despite high fidelity, cannot fully replicate the complexity of real-world environments, leading to potential gaps in generalization.
- The defense method mainly targets known illusion types; its effectiveness against unforeseen or novel illusions remains to be tested.
- Extreme weather conditions and multiple illusions overlapping pose additional challenges that are not fully addressed in current experiments.
Future Work
Future research will focus on integrating multi-sensor data (LiDAR, radar) to improve environmental perception robustness. Developing adaptive models capable of recognizing and compensating for unseen illusions is crucial. Extending benchmarks to include adverse weather and multi-illusion scenarios will enhance real-world applicability. Additionally, exploring end-to-end learning frameworks and industry-standard evaluation protocols will further promote safe deployment.
AI Executive Summary
Autonomous driving systems rely heavily on visual perception to interpret complex traffic environments. However, natural phenomena such as shadows, reflections, and road damages—collectively termed environmental illusions—pose significant challenges by misleading perception models. Despite their prevalence, these illusions have been underexplored in robustness evaluations, leaving safety vulnerabilities unaddressed.
To bridge this gap, the authors introduce LanEvil++, a comprehensive benchmark designed to evaluate the robustness of lane perception models under diverse environmental illusions. Utilizing the CARLA simulator, they generate 94 high-fidelity scenes encompassing 14 illusion types, including shadows, reflections, and road damages, with multiple severity levels. The benchmark provides a large-scale dataset of over 90,000 annotated images, 1,596 video clips, and 41,855 visual question-answer pairs, supporting both traditional lane detection (LD) and vision-language models (ADVLMs).
Extensive experiments reveal that environmental illusions significantly impair model performance. On average, LD models experience a 5.27% accuracy drop, with shadows being the most disruptive, reducing accuracy by up to 7.20%. ADVLMs show a 2.03% decrease in GPT-score, with traffic obstruction illusions causing notable degradation. Closed-loop simulations with systems like OpenPilot demonstrate that illusions can lead to incorrect driving decisions, emphasizing safety concerns. Real-world case studies further validate these vulnerabilities.
To address these issues, the authors propose MIDA, a multimodal illusion defense approach that employs hard sample mining and attention mechanisms. MIDA improves robustness by 4.23% for LD models and 3.82% for ADVLMs, significantly enhancing their resilience against environmental disturbances. This work provides a vital step toward safer autonomous driving by offering a systematic evaluation framework and effective defense strategies. Future efforts will focus on multi-sensor integration, adaptive learning, and expanding benchmarks to encompass more complex scenarios, ultimately aiming to reduce safety risks caused by environmental illusions.
Deep Analysis
Background
Autonomous driving perception技术经历了从传统规则到深度学习的快速演变。早期依赖激光雷达和摄像头，取得一定稳定性，但在复杂环境中的鲁棒性不足。近年来，深度神经网络（如LaneNet、SCNN）推动了车道线检测的突破，但阴影、反射等环境干扰仍是瓶颈。现有数据集（如CULane、BDD-100K）虽丰富，但缺乏针对环境错觉的系统评估工具。行业逐渐认识到环境鲁棒性对安全的重要性，亟需标准化评估体系和针对性解决方案。
Core Problem
环境中的自然错觉如阴影、反射，容易误导感知模型，导致误识别或漏检，威胁驾驶安全。现有模型在标准测试集表现良好，但在实际复杂环境中表现差强人意。缺乏系统性评估和针对性鲁棒性提升策略，使得模型在真实场景中存在巨大安全隐患。如何全面衡量模型对环境错觉的敏感度，并提出有效防御措施，成为行业难题。
Innovation
本研究的创新点包括：1）构建涵盖14类环境错觉的LanEvil++基准，提供大规模、多类别、多场景的评估平台；2）利用CARLA模拟器，系统生成高保真、多级扰动场景，模拟阴影、反射等错觉；3）提出多模态硬样本增强（MIDA），结合模型内部注意力机制，有效提升抗错觉能力；4）在仿真和实地验证中验证方法的有效性，为行业提供技术路径。这些创新突破了传统单一场景、单一模型的局限，为环境鲁棒性研究提供新思路。
Methodology
- �� 设计环境模型：定义静态基础设施、动态对象和环境条件。
- �� 生成错觉场景：在CARLA中模拟阴影、反射、道路裂缝等，设定多级别扰动。
- �� 构建基准：采集94个场景，标注90,292图像，设计6类ADVLM任务，生成丰富测试集。
- �� 模型评估：在不同错觉条件下测试LD和ADVLM模型，指标包括准确率、F1-score、GPT-score。
- �� 提出MIDA：结合硬样本和注意力机制，训练模型增强抗干扰能力。
- �� 仿真验证：在OpenPilot和LMDrive中进行闭环测试，分析驾驶决策变化。
- �� 实地案例：采集真实交通场景，验证模型在实际环境中的表现。
Experiments
采用CARLA模拟环境，生成94个场景，涵盖阴影、反射、道路裂缝等14类错觉。对比多种主流LD模型（如SCNN、LaneATT）和ADVLM（如LMDrive、Dolphins），在正常和错觉条件下评估性能。指标包括准确率、F1-score、GPT-score等。通过不同扰动级别，分析模型性能变化，进行消融实验验证硬样本增强效果。闭环仿真中，结合OpenPilot和LMDrive，观察模型决策变化，实地采集交通场景，验证模型在真实环境中的鲁棒性。
Results
实验显示，阴影错觉导致LD模型平均Accuracy下降5.27%，最高达7.20%；F1-score下降10.49%。ADVLM的GPT-score平均降低2.03%，交通阻塞影响最大，下降3.51%。硬样本增强（MIDA）后，LD性能提升4.23%，ADVLM提升3.82%。闭环仿真中，错觉引发的误判可能导致车辆偏离车道甚至事故。实地案例验证了模型在真实复杂环境中的潜在安全风险，强调鲁棒性提升的必要性。
Applications
该基准和方法可应用于自动驾驶感知系统的鲁棒性评估与优化，帮助企业在模型部署前识别潜在安全隐患。未来可结合多源传感器信息，提升环境感知的抗干扰能力；推动行业标准制定，降低环境错觉引发的交通事故风险。长远来看，有望实现更智能、更安全的自动驾驶系统，普及于城市和高速公路场景。
Limitations & Outlook
模拟环境虽高保真，但仍难完全复制真实交通中的复杂错觉，存在偏差。模型针对特定错觉优化，面对未知或新型错觉时表现仍不足。极端天气、多错觉叠加等场景尚未充分覆盖，模型在极端条件下的鲁棒性仍需提升。未来需结合多模态信息和自适应机制，增强系统的泛化能力和安全性。
Plain Language Accessible to non-experts
想象你在厨房做饭，突然天花板上的灯光投下阴影，或者镜子里的反射让你误以为门外有人。这些自然出现的阴影和反射，就像在自动驾驶中遇到的环境错觉，会让车辆的感知系统迷糊，误判道路和障碍。研究人员就像厨师一样，设计了各种“调料”——不同的错觉场景，测试自动驾驶的“厨艺”。他们发现，这些阴影和反射会让“厨师”——模型——变得不灵光，甚至做出错误的决定。于是，他们开发了“调料包”——MIDA，帮助“厨师”更好地应对这些“调料”，确保“菜肴”——车辆的决策——安全可靠。这个研究就像改良厨艺，让自动驾驶车在各种复杂“厨房”环境中都能稳稳当当地“做菜”。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，突然屏幕上出现阴影或者反光，让你以为前面有障碍物，其实没有。这就像在现实中，自动驾驶的车也会遇到阴影、反射这些“恶作剧”，让它们误以为路上有东西，从而做出错误反应。科学家们就像游戏设计师一样，设计了很多不同的“场景”——比如阴影、反光、道路裂缝——来测试车的“眼睛”。他们发现，这些“恶作剧”会让车变得不靠谱，甚至可能出事故。于是，他们还发明了“超级护盾”——一种让车更聪明的技术，能帮它更好地识别这些“恶作剧”，确保它在复杂环境中也能安全行驶。这个研究就像让赛车变得更聪明、更安全，让它在任何天气和光线条件下都能稳稳当当跑完全程。
Glossary
Environmental Illusion (环境错觉)
自然存在但误导感知的视觉现象，如阴影和反射，影响自动驾驶的感知准确性。
论文中分析阴影、反射等错觉对模型性能的影响。
Lane Detection (车道检测)
识别道路上的车道线和边界的任务，是自动驾驶的基础感知模块。
评估模型在不同错觉场景下的鲁棒性。
ADVLM (Vision-Language Model for AD)
结合视觉信息和自然语言理解的多模态模型，用于增强场景理解和决策能力。
本文测试其在环境错觉中的表现。
CARLA Simulator
开源的自动驾驶仿真平台，用于生成高保真、多类别的交通场景。
用于构建测试场景和生成数据。
MIDA (Multimodal Illusion Defense Approach)
结合硬样本和注意力机制的模型增强策略，提高模型抗错觉能力。
论文提出的核心防御方法。
Open Questions Unanswered questions from this research
- 1 模拟环境虽高保真，但难以完全复制真实交通中的复杂错觉，存在偏差。未来需结合多源传感器信息，提升模型泛化能力。
- 2 对未知或新出现的环境错觉模型适应性不足，需开发更通用的鲁棒性提升技术。
Abstract
Environmental illusions (eg., shadows, reflections, and tire marks) are naturally existing yet overlooked phenomena in real-world driving environments. They can disturb visual perception, leading to misinterpretation of the scene and posing serious safety risks to autonomous driving (AD) systems. However, existing researches largely overlook these phenomena, leaving a critical gap. To address this issue, we study AD robustness through the lane perception perspective, a fundamental task supporting core functions like cruise control and lane centering. We focus on two representative models: conventional lane detection (LD) and vision-language model-based systems (ADVLMs). In this work, we introduce the first benchmark, LanEvil++, for evaluating the robustness of lane perception under environmental illusions. LanEvil++ encompasses 14 types of illusions and leverages the CARLA simulator to generate 94 high-fidelity, fully controllable 3D scenes, yielding a dataset of 90,292 annotated images, 1,596 video clips, and 41,855 visual question answering pairs. Extensive evaluations demonstrate that environmental illusions substantially degrade the performance of state-of-the-art LD methods. On average, LD models experience a 5.27% drop in Accuracy and a 10.49% decline in F1-score, while ADVLMs show a 2.03% reduction in GPT-score and a 0.75% drop in Language-score. Among all illusions, shadows emerge as the most disruptive factor, reducing accuracy by up to 7.20%. Furthermore, closed-loop simulations reveal that these illusions can lead to incorrect driving decisions. Complementary real-world case studies highlight safety-critical failures in actual traffic scenes. To enhance robustness, we propose the Multimodal Illusion Defense Approach (MIDA). MIDA achieves substantial gains under challenging conditions, boosting robustness by 4.23% on LD models and 3.82% on ADVLMs.