DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
DriveSafe combines multimodal scene descriptions with LLMs for risk detection and safety suggestions, outperforming zero-shot models with 52.85% accuracy.
Key Findings
Methodology
DriveSafe employs a multimodal scene understanding framework that integrates spatial, motion, and depth cues to generate detailed, georeferenced scene captions. These captions are constructed using features extracted via Hybrid-Nets, optical flow, and DepthAnything-v2. The system then uses structured prompts to guide a large language model (like LLaMA) for risk assessment and safety suggestion generation. The process involves: • Extracting road/lane segmentation, optical flow, and depth maps; • Generating comprehensive scene descriptions combining global semantics; • Structuring prompts to elicit risk labels, hazard descriptions, and safety advice; • Fine-tuning the LLM with lightweight adapters for domain adaptation. This approach avoids direct end-to-end fine-tuning of the entire multimodal model, improving efficiency and robustness.
Key Results
- On the DRAMA benchmark, DriveSafe-Finetuned achieved a risk detection accuracy of 52.85%, F1 score of 37.15%, surpassing zero-shot performance (23.49% accuracy). Dangerous object localization reached an IoU of 83.0%, with a localization accuracy of 74.8%. Safety suggestion accuracy was 52.85%, F1 37.15%. Ablation studies confirmed the importance of multimodal cues and structured prompts, demonstrating significant performance gains. The model generalizes well across diverse driving scenarios, indicating practical viability.
- The integration of depth, optical flow, and road segmentation features significantly improved risk localization and hazard reasoning. Fine-tuning with lightweight adapters enhanced model stability and domain-specific performance. The results highlight that explicit multimodal scene descriptions effectively guide LLMs to produce accurate risk assessments and actionable safety recommendations, outperforming existing general VLMs.
- The system demonstrates strong potential for real-time autonomous driving applications, providing interpretable risk explanations and safety suggestions that can be integrated into driver assistance modules, ultimately reducing accident rates and improving traffic safety.
Significance
This work advances the field of autonomous vehicle safety by bridging multimodal scene understanding with natural language risk reasoning. It addresses the critical need for interpretable, actionable safety guidance in complex traffic environments. By leveraging structured scene descriptions and domain-specific fine-tuning, DriveSafe significantly improves risk perception accuracy and safety communication, paving the way for more reliable autonomous driving systems. Its approach offers a scalable, efficient solution that can be integrated into real-world systems, enhancing both safety and user trust. The research also opens avenues for further exploration of multimodal language models in safety-critical domains, setting new standards for interpretability and performance.
Technical Contribution
DriveSafe introduces a multimodal scene captioning framework that fuses spatial, motion, and depth cues into detailed, georeferenced descriptions. It employs a structured prompt strategy to guide LLaMA-based models for risk assessment and safety suggestion generation, avoiding costly end-to-end fine-tuning of large models. The key innovations include: • Multi-source feature fusion for comprehensive scene understanding; • Structured prompt design for risk and safety reasoning; • Lightweight adapter-based fine-tuning for domain adaptation. This combination results in a highly efficient, interpretable system capable of delivering accurate risk assessments and safety recommendations, significantly advancing the state-of-the-art in risk-aware scene understanding for autonomous driving.
Novelty
This study is the first to systematically integrate multimodal scene descriptions with structured prompting to perform risk detection and safety suggestion generation in driving scenarios. Unlike prior works that focus solely on risk localization or end-to-end risk prediction, DriveSafe emphasizes interpretability and actionable safety advice. Its novel use of multimodal feature fusion,结合结构化提示引导LLM的策略，突破了传统模型在细粒度风险识别和安全建议方面的限制，提供了一个高效、可扩展的解决方案，为自动驾驶安全感知领域带来了创新突破。
Limitations
- 模型在极端天气或遮挡严重的场景中表现仍有限，主要由于多模态信息的质量受限和场景描述的依赖性；
- 微调过程虽高效，但在超大规模场景下仍需较强算力支持，实时性有待优化；
- 对少见风险类型的识别能力有限，未来需扩展多样化训练数据以提升泛化能力。
Future Work
未来将结合强化学习和持续学习机制，提升模型对新型风险的适应能力。计划引入多智能体协作，优化多车环境中的风险推理与决策，推动自动驾驶系统的安全升级。还将探索多模态信息的更深层次融合与解释能力，增强模型的可解释性和鲁棒性。
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个繁忙的城市街道上开车，周围有很多不同的车辆、行人和交通信号灯。你需要时刻注意这些东西，确保自己和别人都安全。现在，假设你有一个聪明的助手，它可以用多只“眼睛”观察：比如看前面的车是不是慢下来，旁边的行人是不是要过马路，甚至还能感受到距离和速度的变化。这个助手会用简单的话告诉你：“前面有辆车停着，可能会动”，或者“有人在过马路，要小心”。它还能建议你：“减速”或者“绕开这个区域”。这个系统就像DriveSafe一样，结合多种信息，提前帮你发现潜在危险，给出安全建议，让你在路上更放心。
ELI14 Explained like you're 14
想象你在玩一个超级酷的赛车游戏，你要控制你的车在城市街道上跑。游戏里有很多东西，比如快跑的汽车、走路的人、红绿灯。你得随时注意，避免撞到别人。现在，假如你有一个超级厉害的助手，它可以用很多“眼睛”——比如看车的速度、距离、道路情况，然后用简单的话告诉你：“小心，有人在过马路！”或者“前面那辆车停着，可能会动。”这个助手还能建议你：“慢一点”或者“绕开这个区域”。这就像DriveSafe一样，结合多种信息，帮你提前发现危险，给出安全建议，让你在路上跑得更安全、更开心。
Abstract
Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Language Models (MLLMs) have shown promise on general vision-language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural language descriptions. Specifically, our method first generates spatially grounded captions enriched with multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe