MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases
MTRDrive combines memory retrieval and tool interaction, boosting autonomous driving robustness in corner cases by 20% on key metrics.
Key Findings
Methodology
The proposed MTRDrive framework integrates a structured experience retrieval system with dynamic tool usage within a closed-loop reasoning process. It employs CLIP-based visual embeddings to encode scenarios, storing them as structured documents containing scene description, reasoning process, decisions, tools used, and metadata. During inference, the system retrieves relevant past experiences based on semantic similarity, guiding the current decision-making. External tools like object detection and open-vocabulary recognition are invoked to gather environment information. Chain-of-Thought reasoning combined with reinforcement learning (GRPO) optimizes the policy. The training involves a two-stage process: supervised fine-tuning (SFT) to learn basic tool and experience usage, followed by RL-based policy refinement, with explicit rewards for experience relevance and task accuracy.
Key Results
- On NAVSIM, MTRDrive achieves an 88.3 PDMS score, surpassing previous models, with a planning accuracy of 82.6% and overall driving score of 79.8%.
- In zero-shot testing on the new Roadwork-VLM benchmark, the model attains 80.2 driving score, demonstrating strong generalization in unseen construction scenarios.
- Ablation studies confirm that experience retrieval and tool interaction synergistically improve robustness, significantly reducing hallucinations and reasoning errors, especially in out-of-distribution cases.
Significance
This work addresses critical limitations of current vision-language autonomous driving models, notably hallucinations and poor out-of-distribution generalization. By integrating memory-based experience retrieval and environment-aware tool interaction, it enhances decision robustness in complex, unseen scenarios. The approach bridges the gap between perception and reasoning, paving the way for safer, more reliable autonomous vehicles. Its success in challenging Corner Cases signifies a major step toward real-world deployment, offering a scalable framework adaptable to diverse environments and conditions.
Technical Contribution
The paper introduces a novel interactive reasoning architecture combining a structured experience database with dynamic tool invocation, trained via a two-stage process involving supervised fine-tuning and reinforcement learning with explicit format rewards. The experience retrieval mechanism leverages CLIP embeddings for fast, semantic similarity-based search, enabling proactive, context-aware decision-making. The framework's ability to generalize zero-shot to unseen scenarios marks a significant advancement over existing end-to-end models that lack explicit experience reuse and environment interaction, providing a new paradigm for robust autonomous driving.
Novelty
This is the first work to deeply integrate a structured experience retrieval system with proactive tool interaction within a vision-language driving model, emphasizing experience-driven reasoning to improve robustness. Unlike prior methods that rely solely on end-to-end deep learning or chain-of-thought reasoning without external knowledge, this approach actively leverages past experiences and environment tools, significantly enhancing zero-shot generalization in Corner Cases. This represents a fundamental shift toward knowledge-based, interactive autonomous systems.
Limitations
- The reliance on a comprehensive experience database may limit performance in entirely novel, unexperienced scenarios where relevant past experiences are absent.
- Tool detection accuracy depends on external modules, which can introduce errors affecting decision quality.
- Training complexity and computational costs are high, with current models not yet optimized for real-time deployment in resource-constrained environments.
Future Work
Future directions include expanding the experience database with multi-modal data, improving retrieval efficiency, and integrating end-to-end training for real-time deployment. Enhancing robustness against detection errors and extending the framework to multi-agent scenarios are also promising avenues. Further research will explore adaptive experience updating and more sophisticated environment interaction strategies to handle increasingly complex Corner Cases.
AI Executive Summary
Autonomous driving has seen rapid advancements with the emergence of vision-language models (VLMs), which aim to unify perception, reasoning, and control in a single framework. However, despite impressive progress, current models struggle with robustness in complex, unseen scenarios—particularly in Corner Cases such as construction zones or unusual traffic signs. These limitations manifest as hallucinations, misjudgments, and poor generalization, posing safety risks for real-world deployment.
To address these challenges, this paper introduces MTRDrive, a novel framework inspired by human cognition. It combines a structured experience library with proactive experience retrieval and environment-aware tool interaction within a closed-loop reasoning system. The experience library encodes past driving scenarios using CLIP-based semantic embeddings, enabling fast retrieval of relevant experiences. During inference, the model actively retrieves similar past cases, guiding decision-making with contextual knowledge. External tools, such as object detectors and open-vocabulary recognizers, are invoked to gather environment information, which is then integrated into a chain-of-thought reasoning process. This process is trained via a two-stage approach: supervised fine-tuning to learn basic skills, followed by reinforcement learning (GRPO) to optimize decision policies with explicit format rewards.
Experimental results on NAVSIM and the newly constructed Roadwork-VLM benchmark demonstrate the effectiveness of MTRDrive. It achieves an 88.3 PDMS score and 82.6% planning accuracy on NAVSIM, outperforming existing models. In zero-shot scenarios involving complex construction environments, it attains 80.2 driving score, showcasing exceptional generalization. Ablation studies confirm that experience retrieval and tool interaction significantly improve robustness, reducing hallucinations and errors.
This work marks a significant step toward safer, more reliable autonomous vehicles capable of handling real-world Corner Cases. By integrating knowledge reuse with environment interaction, it opens new avenues for research in knowledge-based, interactive autonomous systems. Future efforts will focus on expanding experience databases, optimizing real-time performance, and extending the framework’s applicability to broader, more complex scenarios, ultimately bringing fully autonomous driving closer to reality.
Deep Analysis
Background
近年来，端到端自主驾驶逐渐成为研究焦点，视觉-语言模型（VLMs）如DriveGPT、OpenDriveVLA等在感知与决策中展现潜力。传统方法多依赖模块化设计，存在信息孤岛和误差累积问题。深度学习结合自然语言处理的端到端模型逐步崛起，但在复杂环境和Corner Cases中表现不足，尤其在泛化和可靠性方面存在瓶颈。现有研究多关注模型性能提升，缺乏对未知场景的鲁棒性保障。随着自动驾驶应用的推广，如何在极端复杂环境中确保安全成为核心难题。
Core Problem
当前视觉-语言模型在自主驾驶中的应用面临幻觉、误判和泛化不足等问题，尤其在Corner Cases中表现脆弱。模型难以应对未见场景，导致安全风险增加。解决这些问题需要引入丰富的环境知识和交互机制，提升模型推理能力和环境适应性。如何在保证实时性和准确性的基础上，实现跨场景的鲁棒决策，成为研究的关键难点。现有方法多依赖大规模数据和复杂模型，难以满足实际部署需求。
Innovation
本文提出的创新包括：1）基于结构化经验库的主动检索机制，提升场景理解的语义一致性；2）引入经验驱动的工具调用策略，使模型能主动利用外部检测工具获取关键信息；3）采用两阶段训练（SFT与GRPO强化学习），优化决策策略，增强模型泛化能力。这些创新突破了传统端到端模型的局限，特别是在Corner Cases中的鲁棒性，提供了环境理解与知识复用的新范式。
Methodology
- �� 构建驾驶经验库：利用CLIP视觉编码器对场景进行语义嵌入，存储场景描述、推理路径、决策目标、工具调用和元数据。
- �� 经验检索：推理时，根据场景语义相似度检索相关经验，作为上下文信息。
- �� 工具交互：结合目标检测和开域词汇识别工具主动获取环境信息，指导决策。
- �� 交互推理：利用链式推理，将经验、工具信息和实时感知融合，生成决策方案。
- �� 训练流程：先用SFT进行基础微调，再用GRPO强化学习优化策略，结合格式奖励和任务奖励，提升模型决策质量。
Experiments
在NAVSIM和Roadwork-VLM两个数据集上验证。模型参数为3B，训练采用16块H20 GPU，SFT阶段微调2轮，学习率4e-5，强化学习采用GRPO，KL系数0.02。评估指标包括PDMS、规划准确率和整体驾驶性能。通过消融实验验证经验检索和工具交互的贡献，比较不同模型变体性能差异。模型在NAVSIM上达成88.3 PDMS，规划准确率82.6%，整体指标79.8%；在Roadwork-VLM零-shot测试中表现优异，指标达80.2。
Results
在NAVSIM上，MTRDrive显著优于对比模型，PDMS达88.3，规划准确率82.6%，整体驾驶得分79.8%。在未知施工场景中，零-shot测试取得80.2指标，验证强大泛化能力。消融实验显示，经验检索和工具交互共同作用，显著减少幻觉和推理错误，提升鲁棒性。模型在复杂Corner Cases中的表现优于现有方法，验证了交互推理的有效性。
Applications
该框架适用于高安全性自动驾驶系统，特别是在复杂和未知环境中。通过主动检索和环境交互，提升模型在Corner Cases中的表现，适合智能交通、无人驾驶车辆的实际部署。未来可结合多模态知识图谱，增强环境理解能力，推动自动驾驶向更复杂场景拓展。
Limitations & Outlook
模型对经验库的依赖在极端新颖场景中可能不足，检索不到相关经验时性能下降。工具检测的误差可能影响决策准确性。训练成本较高，尚未实现端到端实时部署。未来需优化检索效率和工具鲁棒性，提升系统的实用性。
Plain Language Accessible to non-experts
想象你在开车，就像在一个复杂的工厂里工作。你不仅要看清楚每个机器和标志，还要记住以前遇到的类似情况，比如前面有施工标志或交通信号灯。每次遇到新情况时，你会回忆起过去的经验，结合当前的环境信息，做出最安全的决策。这个系统就像一个聪明的工人，能主动找出过去的经验，借助工具（比如检测摄像头或标志识别器）帮忙确认环境，然后再决定下一步怎么走。这样，即使遇到以前没见过的复杂场景，也能保证安全行驶。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，你需要在不同的关卡中做出聪明的决定。有时候你会遇到新场景，比如突然出现的障碍物或奇怪的标志。为了应对这些挑战，你会记住以前遇到的类似情况，比如之前遇到过施工区域或交通信号灯，然后用这些经验来帮你做决定。你还可以用特殊的工具，比如放大镜或扫描仪，帮你确认环境。这个系统就像一个聪明的哥哥，他会记住所有的经验，善于用工具，帮你在新场景中找到正确的路。即使遇到从未见过的情况，也能安全顺利地完成任务。
Glossary
视觉-语言模型 (Vision-Language Model, VLM)
一种结合视觉信息和自然语言理解的深度学习模型，用于场景理解和决策支持。
本文中用以实现端到端自主驾驶的场景感知与推理。
经验库 (Experience Base)
存储结构化驾驶场景、推理过程和决策经验的数据库，用于检索相关经验以辅助推理。
模型通过检索经验库提升复杂场景下的鲁棒性。
链式推理 (Chain-of-Thought, CoT)
逐步推导的推理方法，增强模型的可解释性和推理深度。
用于优化自主驾驶中的复杂决策过程。
GRPO强化学习
一种基于策略优化的强化学习算法，用于微调模型决策策略。
训练阶段用以提升模型在复杂场景中的表现。
PDMS (Predictive Driver Model Score)
衡量自主驾驶模型轨迹预测准确性和物理合理性的指标。
评估模型在真实场景中的轨迹预测性能。
Open Questions Unanswered questions from this research
- 1 如何进一步提升经验库的覆盖范围，确保在极端新颖场景中的表现？
- 2 工具检测的误差如何影响整体决策的鲁棒性？
- 3 模型在端到端实时部署中的计算成本和优化策略？
Applications
Immediate Applications
智能驾驶辅助系统
利用MTRDrive提升自动驾驶车辆在复杂环境中的安全性和鲁棒性，适用于高速公路和城市道路。
自动驾驶测试平台
作为仿真和实车测试的核心算法，验证复杂场景下的决策能力，减少事故风险。
Long-term Vision
全自动无人驾驶
实现完全自主的无人驾驶车辆，能在任何环境中安全行驶，推动智能交通普及。
Abstract
Vision-Language Models(VLMs) have demonstrated significant potential for end-to-end autonomous driving, yet a substantial gap remains between their current capabilities and the reliability necessary for real-world deployment. A critical challenge is their fragility, characterized by hallucinations and poor generalization in out-of-distribution (OOD) scenarios. To bridge this gap, we introduce MTRDrive, a novel framework that integrates procedural driving experiences with a dynamic toolkit to enhance generalization and proactive decision-making. MTRDrive addresses these limitations through a closed-loop system that combines a memory-based experience retrieval mechanism with dynamic toolkits. This synergy enables the model to interact more effectively with its environment, improving both reasoning and decision-making capabilities with the help of our memory-tool synergistic reasoning. Additionally, we introduce a new benchmark based on complex Roadwork construction scenarios to rigorously evaluate zero-shot generalization. Extensive experiments demonstrate the superior effectiveness of our approach. On the public NAVSIM benchmark, our 3B-parameter MTRDrive model achieves an exceptional PDMS of 88.3 without chain-of-thought and sets a state-of-the-art performance bar on high-level planning, with a driving metric score of 79.8\% and a planning accuracy of 82.6\%. Rigorous zero-shot evaluation on the new Roadwork-VLM benchmark shows a strong ability to reason robustly in unseen scenarios, achieving a driving metric score of 80.2\%. These results highlight MTRDrive's potential to advance autonomous driving toward safer and more reliable systems.