ADAPT: Action-aware Driving Caption Transformer
ADAPT, a Transformer-based multi-task model, jointly predicts driving actions and generates natural language explanations, achieving CIDEr 34.6 and reasoning 11.4 on BDD-X.
Key Findings
Methodology
ADAPT employs a shared video encoder based on Video Swin Transformer, integrating a vision-language transformer for caption generation and a motion transformer for control prediction. It jointly trains these tasks using a multi-task loss, enabling the model to produce both action descriptions and control signals. The captioning head generates two sentences—action narration and reasoning—using autoregressive sequence modeling with segment embeddings. The control head regresses speed and steering signals via a sequence-to-sequence transformer. The entire framework is optimized end-to-end, leveraging cross-modal attention to enhance feature interaction.
Key Results
- On the BDD-X dataset, ADAPT outperforms previous state-of-the-art methods with CIDEr scores of 34.6 for narration and 11.4 for reasoning, representing over 30% improvements. Human evaluation shows 90% correctness in action descriptions and reasoning accuracy above 90%. Ablation studies confirm the benefits of joint training and multi-modal fusion, with performance gains of approximately 15% over single-task models.
- The system demonstrates real-time deployment capability, processing raw vehicle videos to generate intuitive natural language explanations and control signals, suitable for integration into autonomous vehicles and driver-assist systems.
- Experimental results highlight the importance of dense frame sampling (e.g., 16 frames) and multi-task learning, with the model maintaining robustness across diverse traffic scenarios and weather conditions.
Significance
This work advances the interpretability of autonomous driving systems by translating complex control policies into human-understandable language, fostering greater trust and acceptance. The multi-task framework not only improves explanation quality but also enhances control accuracy, addressing critical safety concerns. Its real-time deployment potential paves the way for more transparent AI in transportation, bridging the gap between technical performance and user comprehension.
Technical Contribution
The paper introduces a novel multi-task Transformer architecture combining video encoding, natural language captioning, and control signal prediction within a unified framework. It innovates with a shared video encoder, segment-aware autoregressive captioning, and cross-modal attention mechanisms, enabling simultaneous action explanation and control prediction. The end-to-end training paradigm ensures task synergy, resulting in superior performance over existing methods that treat these tasks separately. The system's real-time capability demonstrates engineering feasibility for practical deployment.
Novelty
This is the first work to integrate driving action captioning with control signal prediction in a Transformer-based multi-task model, producing natural language explanations directly from raw videos. Unlike prior visual explanation methods, ADAPT offers human-friendly narratives and reasoning, significantly enhancing interpretability. Its joint training approach and real-time deployment distinguish it from existing offline or visual-only interpretability techniques, marking a new direction in explainable autonomous driving.
Limitations
- The model's generalization to highly complex or adverse weather conditions remains limited, primarily due to training data constraints. Its performance drops in scenarios with poor visibility or unusual traffic patterns.
- Real-time deployment requires substantial computational resources, which may hinder widespread adoption on edge devices. Model optimization for efficiency is needed.
- The system currently relies on high-quality front-view videos; occlusions or sensor failures could impair explanation accuracy and control prediction, necessitating multimodal sensor fusion in future work.
Future Work
Future directions include expanding training datasets to cover diverse weather and traffic scenarios, integrating additional sensors like LiDAR and radar for robustness, and applying reinforcement learning to optimize control policies. Further, improving model efficiency for edge deployment and exploring multi-agent interactions for cooperative driving are promising avenues.
AI Executive Summary
Autonomous driving has made significant technological strides, yet its decision-making process remains largely opaque, limiting public trust and regulatory acceptance. Traditional interpretability methods, such as attention maps and cost volumes, provide visual explanations but often fail to communicate complex behaviors intuitively to laypersons. To address this, the paper introduces ADAPT, a Transformer-based multi-task framework that jointly predicts control signals and generates natural language descriptions of driving actions and their underlying reasoning.
ADAPT leverages a shared video encoder based on the Video Swin Transformer, capturing rich spatiotemporal features from raw vehicle front-view videos. Its captioning module employs a vision-language Transformer with segment embeddings to produce two coherent sentences—one describing the vehicle's current action and another explaining the rationale behind it. Simultaneously, a motion transformer predicts control signals like speed and steering, enabling the system to perform both tasks in an end-to-end manner.
Experimental results on the large-scale BDD-X dataset demonstrate that ADAPT surpasses existing methods significantly, achieving CIDEr scores of 34.6 for narration and 11.4 for reasoning, with human evaluations confirming high correctness rates. The system's real-time deployment capability was validated in simulated and real-world scenarios, where it provided clear, understandable explanations of vehicle behaviors, fostering transparency.
This work marks a breakthrough in explainable autonomous driving, bridging the gap between complex control policies and human comprehension. Its multi-task architecture not only improves interpretability but also enhances control accuracy, promising safer and more trustworthy autonomous vehicles. Future research will focus on expanding data diversity, optimizing computational efficiency, and integrating multimodal sensors to further improve robustness and applicability in diverse traffic environments.
Deep Dive
Abstract
End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practice. There have been some early attempts to use attention maps or cost volume for better model explainability which is difficult for ordinary passengers to understand. To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. ADAPT jointly trains both the driving caption task and the vehicular control prediction task, through a shared video representation. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time. The code, models and data are available at https://github.com/jxbbb/ADAPT.