Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving
Senna integrates LVLM with end-to-end models, reducing planning error by 27.12% and collision rate by 33.33%.
Key Findings
Methodology
Senna combines Senna-VLM and Senna-E2E. Senna-VLM uses multi-view image encoding and prompts, leveraging pre-trained reasoning to produce natural language high-level decisions. Senna-E2E predicts precise trajectories conditioned on these decisions. A three-stage training (pretraining, driving fine-tuning, planning fine-tuning) enhances performance. Planning-oriented QAs and auto-labeled large-scale data facilitate training. Experiments on DriveX and nuScenes show state-of-the-art results, with a 27.12% reduction in average planning error and a 33.33% collision rate decrease after pretraining and fine-tuning.
Key Results
- On DriveX validation, Senna achieves a planning F1 score of 71.21%, outperforming LLaVA-1.5 by about 7 percentage points.
- On nuScenes, the average L2 trajectory error is 0.43 meters, with a collision rate of 0.12%, lower than baseline models.
- Pretraining on DriveX followed by nuScenes fine-tuning reduces error by 27.12% and collision rate by 33.33%, demonstrating strong transferability.
Significance
This work bridges reasoning and precise trajectory prediction, significantly advancing autonomous driving safety and robustness. It introduces a structured, interpretable decision-making framework that leverages multimodal reasoning, addressing long-standing challenges in complex scenarios. The approach enhances generalization across diverse environments, paving the way for fully autonomous vehicles with explainable AI components.
Technical Contribution
The paper proposes a hierarchical decision framework, decoupling high-level natural language decisions from low-level trajectory prediction. It employs multi-view encoding and dedicated prompts for scene understanding, combined with a novel three-stage training strategy that integrates auto-labeled data and reasoning tasks. This design surpasses prior methods like DriveVLM and VAD in accuracy and interpretability, offering new theoretical insights and practical engineering solutions for multimodal autonomous driving.
Novelty
This is the first work to embed natural language reasoning directly into high-level planning for autonomous vehicles, avoiding the limitations of LVLMs in numerical prediction. The multi-view prompts and structured decision pipeline significantly improve scene understanding and decision coherence, setting a new standard over prior end-to-end or LVLM-based approaches.
Limitations
- The system's performance drops in adverse weather or occlusion scenarios, where perception becomes unreliable. The reliance on large-scale pretraining increases computational costs, limiting real-time deployment. Ambiguities in natural language decisions may lead to inconsistent behaviors, requiring further refinement. Future work should focus on robustness, efficiency, and safety guarantees under diverse conditions.
Future Work
Future directions include integrating reinforcement learning to optimize decision policies, enhancing multimodal fusion for robustness, and reducing computational overhead. Developing explainability modules and user interaction interfaces will improve trust and safety. Expanding training datasets with diverse weather and traffic conditions will further strengthen generalization.
AI Executive Summary
Autonomous driving has rapidly evolved, yet existing solutions struggle with complex, rare scenarios due to limited reasoning and interpretability. End-to-end models excel at trajectory prediction but lack high-level decision-making and common sense, often leading to errors in unpredictable situations. Conversely, large vision-language models (LVLMs) demonstrate remarkable scene understanding and reasoning but are not suited for precise numerical predictions like trajectories.
This paper introduces Senna, a structured autonomous driving system that bridges these gaps by integrating LVLMs with end-to-end trajectory prediction models. Senna comprises two core modules: Senna-VLM, which encodes multi-view images and generates high-level natural language planning decisions, and Senna-E2E, which predicts precise trajectories conditioned on these decisions. The system employs multi-view prompts, a dedicated driving vision adapter, and a series of planning-oriented question-answering tasks to enhance scene understanding and decision accuracy.
The innovative approach involves a three-stage training strategy—mixed pretraining, driving fine-tuning, and planning fine-tuning—allowing the model to leverage large-scale datasets like DriveX and nuScenes effectively. Extensive experiments demonstrate that Senna achieves state-of-the-art planning performance, reducing average planning error by 27.12% and collision rate by 33.33% after pretraining and fine-tuning. The system's hierarchical decision-making process, combining reasoning and precise control, offers interpretability and robustness, essential for real-world deployment.
This work marks a significant step toward fully autonomous vehicles, providing a framework that combines the reasoning prowess of LVLMs with the accuracy of end-to-end models. Despite current limitations in adverse weather and computational costs, the proposed architecture opens new avenues for research in multimodal, explainable, and scalable autonomous driving systems.
Deep Analysis
Background
Autonomous driving技术经历了从模块化感知、预测到端到端学习的演变。早期系统依赖规则和手工设计，存在泛化能力不足的问题。近年来，深度学习推动端到端模型崛起，如Uber的PilotNet和Waymo的VAE，显著提升了感知和规划性能。大规模数据和Transformer架构的引入，使模型在复杂场景中表现更优，但仍缺乏良好的推理和解释能力。LVLM的出现为场景理解带来新机遇，结合视觉和语言，增强模型的常识推理和多任务能力。DriveVLM等工作尝试将LVLM应用于自主驾驶，但在数值预测方面仍存在局限。本文在此基础上，提出结构化决策与多视角编码，推动了多模态自主驾驶的研究前沿。
Core Problem
现有端到端模型在复杂、罕见场景中表现不足，主要因缺乏常识推理能力，难以应对突发情况。直接预测轨迹或控制信号，缺少高层决策解释，易受噪声干扰。LVLM虽具推理优势，但难以进行精确数值预测，导致轨迹误差较大。如何融合两者优势，提升场景理解和决策合理性，成为核心难题。特别是在多视角、多模态信息融合、自然语言决策生成、训练数据标注等方面，仍有较大挑战。
Innovation
提出Senna系统，创新点包括：1）将LVLM用于高层决策生成，利用其推理能力避免数值预测误差；2）多视角图像编码和专用提示，增强空间理解；3）三阶段训练策略，结合自动问答提升场景理解；4）结构化决策流程，将自然语言决策转化为轨迹预测条件。此设计突破了传统端到端模型的局限，为自主驾驶提供了更具解释性和鲁棒性的解决方案。
Methodology
- �� 输入多视角图像、用户指令和导航信息。• Senna-VLM采用ViT-L/14编码图像，Driving Vision Adapter压缩特征，生成图像Token。• 文本编码器处理指令，结合多视角提示，输入到大模型（如Vicuna-v1.5-7b）中。• 生成高层次决策（自然语言），通过Meta-action Encoder转为特征。• Senna-E2E根据场景和高层决策，预测精确轨迹。• 训练采用三阶段：混合预训练、驾驶微调、规划微调，结合自动问答增强理解。
Experiments
在DriveX和nuScenes数据集上进行评估，比较多种模型指标。DriveX验证集上，Senna规划F1达71.21%，优于LLaVA-1.5。nuScenes上，平均L2误差为0.43米，碰撞率0.12%。通过预训练微调，误差降低27.12%，碰撞率降低33.33%。采用ablation验证多视角编码、问答策略和训练阶段的贡献，显示模型在复杂场景中的优越表现。
Results
Senna在多项指标中优于对比模型，尤其在规划误差和安全性方面表现突出。预训练后微调显著提升泛化能力，验证其跨场景适应性。模型在复杂交通环境中表现出较强鲁棒性，验证了结构化决策和多模态融合的有效性。
Applications
可应用于自动驾驶车辆的高层决策支持，提升在复杂交通环境中的安全性和鲁棒性。适合未来智能交通系统、自动驾驶平台集成，尤其在多视角感知和自然语言交互方面具有优势。长远来看，有助于实现全自动驾驶的普及和商业化。
Limitations & Outlook
模型在极端天气、遮挡和突发事件中仍存在识别和决策误差。对大规模预训练数据依赖较强，训练成本高，实时性待提升。自然语言决策可能引入歧义，需进一步优化解释一致性和安全性。未来需结合强化学习和多模态优化，增强系统鲁棒性。
Plain Language Accessible to non-experts
想象你在开车，面对复杂的交通场景，比如红绿灯、行人、其他车辆。传统方法就像用一张地图逐步规划路线，但遇到突发情况就会迷糊。Senna系统像是有一个聪明的助手，它能用“语言”告诉你：前方红灯，慢下来，右转避让行人。这个助手还会用多只眼睛看四周，理解环境，就像你用多只眼睛看世界一样。然后，系统会根据这些“指令”帮你画出最安全的行驶轨迹。这样，车子不仅能快速反应，还能理解为什么要这么做，就像和你讲故事一样。这个方法让自动驾驶变得更聪明、更安全，也更像人类驾驶员的思考方式。
ELI14 Explained like you're 14
想象你在开车，路上有很多不同的情况，比如红灯、行人、转弯。以前的自动驾驶就像用一张简单的地图，遇到复杂情况就容易出错。现在，Senna就像有一个特别聪明的朋友，他会用“话”告诉你：前面红灯，慢点，右转避让行人。这个朋友还能用多只眼睛看四周，理解环境，就像你用多只眼睛看世界一样。然后，系统会根据这些“话”和“眼睛”看到的东西，帮你画出一条最安全的路线。这样，车子不仅反应快，还能理解为什么要这么做，就像和你讲故事一样。这个新方法让自动驾驶变得更聪明、更安全，也更像人类驾驶员的思考方式。
Glossary
Large Vision-Language Model (LVLM) (大规模视觉-语言模型)
结合视觉和语言信息，具有推理和理解能力的深度学习模型。用于场景理解和决策生成。
Senna中的Senna-VLM利用LVLM进行高层次决策。
Meta-action (元动作)
高层次的驾驶决策指令，如转弯、加速，作为轨迹预测的条件。
作为Senna系统中高层决策的表达形式。
三阶段训练策略 (Three-stage training strategy)
包括混合预训练、驾驶微调和规划微调，逐步优化模型性能。
提升Senna在复杂场景中的规划能力。
多视角编码 (Multi-view encoding)
整合来自多个角度的图像信息，增强空间理解。
Senna-VLM采用此技术理解全景场景。
自动问答 (Auto-labeled QAs)
通过自动化流程生成的场景理解和规划相关问答，用于训练。
增强模型对场景的理解和决策能力。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端天气和遮挡条件下的感知和决策能力，仍是未来研究重点。
- 2 多模态融合的实时优化和模型压缩技术，需解决以实现高效部署。
- 3 自然语言决策的歧义处理和安全性保障，是系统推广的关键难题。
Applications
Immediate Applications
智能驾驶辅助系统
集成Senna提升自动驾驶车辆在复杂交通环境中的决策能力，增强安全性和鲁棒性。
自动驾驶平台优化
为自动驾驶公司提供高层决策支持，改善路径规划和环境理解。
Long-term Vision
全自动驾驶普及
推动Senna技术在商业化自动驾驶车辆中的应用，实现无人驾驶普及。
Abstract
End-to-end autonomous driving demonstrates strong planning capabilities with large-scale data but still struggles in complex, rare scenarios due to limited commonsense. In contrast, Large Vision-Language Models (LVLMs) excel in scene understanding and reasoning. The path forward lies in merging the strengths of both approaches. Previous methods using LVLMs to predict trajectories or control signals yield suboptimal results, as LVLMs are not well-suited for precise numerical predictions. This paper presents Senna, an autonomous driving system combining an LVLM (Senna-VLM) with an end-to-end model (Senna-E2E). Senna decouples high-level planning from low-level trajectory prediction. Senna-VLM generates planning decisions in natural language, while Senna-E2E predicts precise trajectories. Senna-VLM utilizes a multi-image encoding approach and multi-view prompts for efficient scene understanding. Besides, we introduce planning-oriented QAs alongside a three-stage training strategy, which enhances Senna-VLM's planning performance while preserving commonsense. Extensive experiments on two datasets show that Senna achieves state-of-the-art planning performance. Notably, with pre-training on a large-scale dataset DriveX and fine-tuning on nuScenes, Senna significantly reduces average planning error by 27.12% and collision rate by 33.33% over model without pre-training. We believe Senna's cross-scenario generalization and transferability are essential for achieving fully autonomous driving. Code and models will be released at https://github.com/hustvl/Senna.