Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability

Autonomous driving has seen rapid advances through end-to-end deep learning models, yet challenges in generalization and interpretability persist. Traditional models often perform well in training environments but falter in unseen scenarios, limiting safety and deployment. Moreover, their black-box nature hampers understanding of failure causes, impeding trust and regulatory approval. This paper introduces a novel framework that incorporates multi-task perception—specifically segmentation and depth estimation—as intermediate representations within an end-to-end driving system. The perception module, based on residual networks with hybrid dilated convolutions, learns pixel-level scene understanding, providing 'what & where' and 'how far' knowledge. These outputs serve as interpretable cues, enabling visualization of the model’s perception and diagnosis of failures. The driving module, built with residual blocks, utilizes these cues to generate control commands for various high-level guidance signals. The training employs a stepwise strategy: first, the perception module learns multi-task knowledge; then, its parameters are frozen while the driving module is trained. Extensive experiments in CARLA simulator demonstrate that this approach achieves a 20% success rate increase in untrained environments, outperforming existing methods. The visualization of segmentation and depth maps allows practitioners to trace back the cause of errors, significantly enhancing safety and reliability. This work not only advances the state-of-the-art in autonomous driving robustness but also sets a new standard for model interpretability. Future work aims to incorporate multi-modal sensors, optimize real-time performance, and validate on real-world roads, moving closer to safe, explainable autonomous vehicles.