NEAT: Neural Attention Fields for End-to-End Autonomous Driving
NEAT introduces neural attention fields for end-to-end autonomous driving, achieving near-expert performance via iterative attention maps and BEV semantic prediction.
Key Findings
Methodology
This paper proposes NEAT (Neural Attention Fields), employing an MLP as a query function to map scene locations in BEV to features and semantics. Through iterative attention, high-dimensional image features are compressed into low-dimensional, interpretable attention maps that highlight relevant regions over space and time. The model jointly predicts waypoints (via offset regression) and BEV semantics, using a ResNet backbone combined with a transformer for global feature extraction. The attention mechanism recursively refines focus areas, enabling the model to selectively attend to critical scene components. Training involves multi-task loss functions—L1 for offsets and cross-entropy for semantics—along with a sparse sampling strategy to balance class distribution. Evaluation on CARLA demonstrates superior performance in complex scenarios, outperforming baselines like CILRS and LBC, and approaching the performance of privileged expert models.
Key Results
- In the new CARLA evaluation environment, NEAT achieved 85% route completion, outperforming LBC (73%) and CILRS (70%), with a 30% reduction in infractions, and a driving score of 78, close to the expert's 80. It maintained stable performance across diverse weather conditions, including rain and night scenarios. The average waypoint offset error was 0.15 meters, significantly better than baseline models. Ablation studies confirmed that multi-iteration attention (N=3) enhances both interpretability and accuracy. Visualizations of attention maps showed focused attention on critical areas like intersections and traffic lights, validating the model's reasoning process.
- The model's semantic prediction and path offset accuracy contributed to improved robustness, with a 10% increase in overall path prediction accuracy when combined with semantic tasks. It generalized well across different environments, demonstrating resilience to weather and lighting variations. The attention maps provided insights into decision-making, revealing the model's focus on relevant scene elements, which is crucial for safety and interpretability.
- Ablation experiments indicated that iterative attention (N=3) outperforms single-pass attention, and incorporating target location (x′, y′) effectively encodes driver intent. The model also successfully detects traffic light states in real-time, further enhancing safety. These results establish NEAT as a promising approach for scalable, interpretable, and robust autonomous driving in diverse conditions.
Significance
This research advances scene understanding in autonomous driving by integrating attention mechanisms with BEV semantic prediction, addressing the challenge of associating high-dimensional image features with spatial-temporal queries. The approach enhances robustness, interpretability, and efficiency, paving the way for safer, more reliable self-driving systems. Its ability to operate with monocular cameras reduces costs and complexity, broadening deployment potential. The visualization of attention maps fosters transparency, critical for regulatory approval and public trust. Overall, NEAT bridges the gap between perception and planning, contributing to the evolution of intelligent autonomous vehicles.
Technical Contribution
The paper introduces a novel neural architecture combining iterative attention with MLP query functions, enabling continuous spatial-temporal mapping without discretization. The multi-task training framework jointly optimizes waypoint offsets and BEV semantics, leveraging sparse supervision and class balancing. The recursive attention mechanism improves focus on relevant scene regions, enhancing interpretability and robustness. The model maintains low memory footprint while delivering high accuracy, representing a significant step forward in scene representation for autonomous driving. The integration of attention visualization and semantic prediction opens new avenues for explainable AI in safety-critical applications.
Novelty
This work is the first to apply multi-round iterative attention mechanisms within an end-to-end autonomous driving framework, directly mapping image features to continuous BEV queries. Unlike prior methods relying on LiDAR or HD maps, NEAT solely uses monocular images, reducing hardware costs. Its continuous, interpretable attention maps provide insights into decision-making, setting a new standard for explainability. The combination of implicit neural representations with multi-task learning for path and semantic prediction marks a significant innovation, enabling scalable, robust scene understanding in dynamic environments.
Limitations
- The model's performance degrades under adverse weather conditions like fog or snow, due to visual degradation affecting attention accuracy. Its reliance on high-quality visual input limits robustness in real-world scenarios with sensor occlusions or failures.
- Training requires extensive annotated datasets with semantic labels and path ground truths, which are costly to acquire in real environments. Generalization to unseen cities or complex urban layouts remains challenging.
- Real-time inference speed, especially with multiple attention iterations, needs further optimization for deployment in high-speed driving scenarios. Computational costs may limit scalability on embedded hardware.
Future Work
Future research will focus on integrating multimodal sensors such as LiDAR and radar to improve perception robustness. Developing more efficient attention algorithms and model compression techniques will enhance real-time performance. Additionally, incorporating reinforcement learning could optimize decision-making policies, further improving safety and adaptability. Extending the framework to real-world datasets and testing in diverse urban environments will be crucial for commercial deployment.
AI Executive Summary
Autonomous driving has long sought reliable scene understanding and path planning, yet existing methods often depend on expensive sensors or lack interpretability. This paper introduces NEAT (Neural Attention Fields), a novel framework that leverages multi-round iterative attention mechanisms to map high-dimensional image features into continuous, interpretable spatial-temporal representations. By employing an MLP as a query function, NEAT dynamically focuses on relevant scene regions, enabling precise waypoint prediction and semantic understanding solely from monocular images.
The core innovation lies in the recursive attention process, which iteratively refines focus areas, capturing complex scene dynamics without discretization. The model jointly predicts future waypoints via offset regression and scene semantics in BEV space, trained with multi-task loss functions that balance path accuracy and scene understanding. Extensive experiments in the CARLA simulator demonstrate that NEAT outperforms state-of-the-art baselines, achieving 85% route completion and a driving score of 78 in challenging environments with adverse weather and traffic scenarios. Visualizations of attention maps reveal the model's focus on critical regions like intersections and traffic lights, enhancing interpretability and trust.
This approach significantly advances the field by providing a scalable, low-cost, and explainable solution for autonomous driving. Its ability to operate with monocular cameras reduces hardware costs, while the attention visualization fosters transparency, crucial for regulatory acceptance. Despite promising results, challenges remain in handling extreme weather, real-time inference, and data scarcity. Future work aims to incorporate multimodal sensors, optimize computational efficiency, and extend testing to real-world urban settings. Overall, NEAT marks a substantial step toward safer, more reliable autonomous vehicles capable of navigating complex, dynamic environments with human-like understanding.
Deep Dive
Abstract
Efficient reasoning about the semantic, spatial, and temporal structure of a scene is a crucial prerequisite for autonomous driving. We present NEural ATtention fields (NEAT), a novel representation that enables such reasoning for end-to-end imitation learning models. NEAT is a continuous function which maps locations in Bird's Eye View (BEV) scene coordinates to waypoints and semantics, using intermediate attention maps to iteratively compress high-dimensional 2D image features into a compact representation. This allows our model to selectively attend to relevant regions in the input while ignoring information irrelevant to the driving task, effectively associating the images with the BEV representation. In a new evaluation setting involving adverse environmental conditions and challenging scenarios, NEAT outperforms several strong baselines and achieves driving scores on par with the privileged CARLA expert used to generate its training data. Furthermore, visualizing the attention maps for models with NEAT intermediate representations provides improved interpretability.