Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
This study questions whether ego status alone suffices for open-loop end-to-end autonomous driving, revealing dataset bias and metric limitations that favor ego-only reliance.
Key Findings
Methodology
The authors reproduce the AD-MLP model, removing historical trajectory inputs, relying solely on ego status (velocity, yaw, etc.) for trajectory prediction. They introduce a new road deviation metric to assess trajectory safety. Multiple models, including pure ego-based and perception-fused variants, are compared using L2 distance, collision rate, and the new metric. The analysis reveals over-reliance on ego status driven by dataset bias and evaluation shortcomings. A simple baseline, BEV-Planner, demonstrates competitive performance without perception annotations, challenging assumptions about perception necessity.
Key Results
- Pure ego status models like Ego-MLP achieve comparable results to complex perception models on nuScenes, with L2 distances around 0.35 meters and collision rates below 1%. The new road deviation metric shows frequent off-road trajectories, indicating safety concerns. Perturbation tests reveal high sensitivity to ego velocity changes, raising safety issues. The simple BEV-Planner, relying only on ego status, performs similarly in L2 and collision metrics, questioning the added value of perception inputs in certain scenarios.
- Comparison across models shows reliance on ego status dominates path prediction accuracy. Removing perception inputs causes minimal performance drop, emphasizing data bias effects. Experiments perturb ego velocity and perception inputs, demonstrating that models are disproportionately influenced by ego status, which could lead to unsafe behaviors. The findings suggest current benchmarks and metrics may overstate the importance of perception in straightforward driving scenarios.
- Overall, the study highlights that dataset bias and inadequate metrics can mislead research conclusions, promoting overconfidence in ego-only models. It advocates for more diverse datasets and comprehensive evaluation metrics to better reflect real-world complexity, ultimately guiding the development of safer, more robust autonomous driving systems.
Significance
This research critically examines the reliance on ego status in current autonomous driving models, exposing how dataset biases and limited metrics can produce misleadingly optimistic results. It underscores the importance of diversified data and more holistic evaluation frameworks to ensure safety and robustness in real-world scenarios. For industry, it calls for caution against overfitting to simple benchmarks, emphasizing the need for models that incorporate richer environmental understanding. Academically, it encourages rethinking the role of perception versus ego state, fostering more balanced, generalizable solutions. The insights could influence future dataset design, evaluation standards, and model architectures, ultimately advancing toward safer autonomous vehicles.
Technical Contribution
The paper introduces a minimalist trajectory prediction baseline, Ego-MLP, relying solely on ego status, and demonstrates its competitive performance on nuScenes. It proposes a new road deviation metric to better evaluate trajectory safety, addressing the limitations of traditional L2 and collision metrics. The authors systematically analyze the bias introduced by dataset composition and evaluation metrics, revealing how simple ego-based models can outperform perception-dependent approaches in certain contexts. The work advocates for re-evaluating current benchmarks and encourages the development of more comprehensive, bias-resistant evaluation methods, influencing future research directions in end-to-end autonomous driving.
Novelty
This is the first comprehensive analysis showing that dataset bias—specifically the prevalence of straightforward, ego-dependent scenarios—can lead models to over-rely on ego status. It challenges the assumption that perception is always necessary for safe path planning, supported by a new road deviation metric that highlights safety risks overlooked by traditional metrics. The work also demonstrates that a simple, perception-free baseline can achieve state-of-the-art results in certain benchmarks, prompting a re-examination of evaluation practices and the true contribution of perception modules.
Limitations
- The analysis primarily focuses on nuScenes, which has a strong bias toward straightforward driving scenarios, limiting the generalizability of conclusions to more complex environments. The new road deviation metric, while addressing some safety concerns, does not encompass all safety-critical factors like dynamic obstacle interactions. The models tested are relatively simple and may not reflect the performance of more sophisticated perception-based systems in diverse real-world conditions. Further research is needed to validate these findings across varied datasets and in closed-loop settings.
Future Work
Future research should incorporate more diverse, challenging datasets to mitigate bias. Developing comprehensive evaluation metrics that better reflect safety and robustness is crucial. Exploring multi-source perception integration, especially in complex scenarios, will enhance model reliability. Additionally, establishing real-world closed-loop testing platforms will validate model performance beyond static benchmarks. The community should also investigate how to balance ego state reliance with environmental understanding, ensuring safer and more adaptable autonomous systems.
AI Executive Summary
Autonomous driving research has seen rapid advances through end-to-end learning models that aim to unify perception and planning. However, recent studies reveal that many models achieve high performance primarily due to dataset biases and inadequate evaluation metrics. In particular, the nuScenes dataset, widely used for open-loop evaluation, predominantly features straightforward, straight-line driving scenarios. This bias enables models relying solely on ego vehicle status—such as velocity and yaw—to perform remarkably well, often matching or surpassing perception-based models in metrics like L2 distance and collision rate.
The authors reproduce a simple model, Ego-MLP, which predicts trajectories based only on ego status, and find it performs comparably to complex models that incorporate perception inputs. They introduce a new evaluation metric—road boundary deviation rate—to better assess safety and path correctness, revealing that many models tend to generate trajectories that deviate from the road more frequently than metrics suggest. Further analysis shows that models are highly sensitive to ego velocity perturbations, raising safety concerns.
This work critically questions the current benchmarking practices, emphasizing that dataset bias and limited metrics can mislead the community into overestimating perception’s role. The findings suggest that, in simple scenarios, ego status alone can suffice, but this does not generalize to complex environments. The study advocates for diversified datasets and more comprehensive evaluation frameworks to ensure models are truly robust and safe in real-world conditions.
Overall, the paper calls for a reassessment of current research directions, highlighting the importance of balanced perception and ego information, improved metrics, and realistic testing environments. These insights aim to steer autonomous driving development toward safer, more reliable systems capable of handling the complexities of real-world traffic.
Deep Dive
Abstract
End-to-end autonomous driving recently emerged as a promising research direction to target autonomy from a full-stack perspective. Along this line, many of the latest works follow an open-loop evaluation setting on nuScenes to study the planning behavior. In this paper, we delve deeper into the problem by conducting thorough analyses and demystifying more devils in the details. We initially observed that the nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models incorporating ego status, such as the ego vehicle's velocity. These models tend to rely predominantly on the ego vehicle's status for future path planning. Beyond the limitations of the dataset, we also note that current metrics do not comprehensively assess the planning quality, leading to potentially biased conclusions drawn from existing benchmarks. To address this issue, we introduce a new metric to evaluate whether the predicted trajectories adhere to the road. We further propose a simple baseline able to achieve competitive results without relying on perception annotations. Given the current limitations on the benchmark and metrics, we suggest the community reassess relevant prevailing research and be cautious whether the continued pursuit of state-of-the-art would yield convincing and universal conclusions. Code and models are available at \url{https://github.com/NVlabs/BEV-Planner}