Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks
Proposes heterogeneous auxiliary network feature mimicking, improving steering prediction MAE by 12.8%/52.1% on Udacity/Comma.ai datasets.
Key Findings
Methodology
This paper introduces multi-layer feature mimicking from heterogeneous auxiliary networks such as PSPNet and FlowNet2 to enhance end-to-end steering prediction. The approach involves extracting deep features at multiple layers, transforming them via dedicated layers to reduce dimensionality while preserving spatial information, and then applying L2 loss to align these features with the main network during training. A multi-stage training process is employed: initial optimization of steering prediction, followed by feature mimicking, ensuring stable convergence. The main network, a 50-layer 3D ResNet combined with LSTM, is trained on datasets like Udacity and Comma.ai, achieving significant performance gains, with MAE dropping to 1.62 and 0.70 respectively, and RMSE to 2.35 and 0.98.
Key Results
- On Udacity, the proposed model reduces MAE from 2.56 (3D CNN+LSTM baseline) to 1.62, a 36% improvement; on Comma.ai, MAE drops from 1.14 to 0.70, a 38% gain.
- Feature mimicking from heterogeneous networks improves robustness in challenging scenarios like sharp turns and lighting changes.
- Multi-layer mimicking, especially at high-level features, yields the largest performance boost.
Significance
This work addresses the key limitation of relying solely on steering angles, which lack rich environmental context. By leveraging pre-trained networks for scene segmentation and optical flow, it enriches the model's understanding of scene structure and motion without extra annotations. The approach enhances robustness and generalization, crucial for real-world autonomous driving, and reduces reliance on costly data labeling. It bridges the gap between simple end-to-end models and complex multi-task systems, offering a scalable, effective solution for safer autonomous vehicles.
Technical Contribution
The paper pioneers the integration of heterogeneous deep features into steering prediction via multi-layer feature mimicking, introducing transformation layers for effective feature alignment. It employs a 50-layer 3D ResNet initialized from ImageNet, combined with multi-stage training, to achieve superior accuracy. The method avoids additional labeling costs by utilizing off-the-shelf networks, enabling rich contextual learning. This framework opens new avenues for multi-source feature fusion in end-to-end driving models, setting a new state-of-the-art.
Novelty
This is the first systematic use of heterogeneous auxiliary network features (e.g., segmentation and optical flow) for steering angle prediction, moving beyond traditional multi-task learning. The multi-layer mimicking strategy, combined with transformation layers, effectively fuses multi-scale, multi-source deep features, providing richer scene understanding. Unlike prior work limited to single-source or single-layer features, this approach demonstrates significant performance gains and robustness improvements.
Limitations
- Despite improvements, the model still struggles under extreme lighting or highly dynamic scenarios where auxiliary networks' generalization is limited.
- Training complexity increases due to multi-stage optimization and multiple auxiliary networks, demanding substantial computational resources.
- Current framework does not incorporate multi-modal sensors like LiDAR or radar, which could further enhance robustness in adverse conditions.
Future Work
Future research will explore integrating multi-modal sensor data, simplifying training procedures, and developing online adaptation mechanisms. Extending the framework to real-time online learning and broader scene understanding will further improve robustness and deployment feasibility in diverse environments.
AI Executive Summary
Autonomous driving systems have rapidly evolved from modular pipelines to end-to-end deep learning models. However, most existing approaches rely solely on steering angles as supervision, limiting their ability to understand complex scenes. These models often falter in scenarios involving sharp turns, shadows, or challenging traffic conditions, due to insufficient contextual information. To address this, the paper introduces a novel training paradigm—heterogeneous auxiliary network feature mimicking—that leverages pre-trained networks like PSPNet for scene segmentation and FlowNet2 for optical flow estimation. During training, deep features from these networks are extracted at multiple layers, transformed via dedicated layers to align dimensions, and then mimicked by the main steering network using L2 loss. This multi-layer, multi-source feature alignment enriches the model’s environmental understanding without requiring additional annotations for the auxiliary tasks. The core model, a 50-layer 3D ResNet combined with LSTM, is trained in a multi-stage process, first optimizing basic steering prediction, then progressively incorporating feature mimicking. Extensive experiments on Udacity and Comma.ai datasets demonstrate that this approach reduces MAE by over 12% and 52%, respectively, outperforming prior state-of-the-art methods. The model exhibits superior robustness in complex scenarios, such as sharp curves and lighting variations, indicating significant potential for real-world deployment. This method’s key advantage is its ability to utilize existing powerful networks as knowledge sources, avoiding costly data labeling while substantially improving prediction accuracy. Looking ahead, integrating multi-modal sensors and online learning strategies could further elevate autonomous driving safety and reliability, making this a promising direction for future research.
Deep Dive
Abstract
The training of many existing end-to-end steering angle prediction models heavily relies on steering angles as the supervisory signal. Without learning from much richer contexts, these methods are susceptible to the presence of sharp road curves, challenging traffic conditions, strong shadows, and severe lighting changes. In this paper, we considerably improve the accuracy and robustness of predictions through heterogeneous auxiliary networks feature mimicking, a new and effective training method that provides us with much richer contextual signals apart from steering direction. Specifically, we train our steering angle predictive model by distilling multi-layer knowledge from multiple heterogeneous auxiliary networks that perform related but different tasks, e.g., image segmentation or optical flow estimation. As opposed to multi-task learning, our method does not require expensive annotations of related tasks on the target set. This is made possible by applying contemporary off-the-shelf networks on the target set and mimicking their features in different layers after transformation. The auxiliary networks are discarded after training without affecting the runtime efficiency of our model. Our approach achieves a new state-of-the-art on Udacity and Comma.ai, outperforming the previous best by a large margin of 12.8% and 52.1%, respectively. Encouraging results are also shown on Berkeley Deep Drive (BDD) dataset.