A DVDrive Approach for doScenes Instructed Driving Challenge
DVDrive introduces divided-view perception with Transformer, improving instruction-conditioned trajectory prediction, reducing ADE from 0.372 to 0.358.
Key Findings
Methodology
Building on OmniDrive, the approach uses nuScenes data, extending trajectory prediction to 6 seconds with 12 waypoints. It incorporates a DVPE-inspired divided-view perception module, dividing multi-view features into local virtual views based on 3D spatial locations. Queries and image tokens are transformed into local coordinate systems, enabling visibility-aware cross-attention within each view. The model fuses visual and language inputs, leveraging Transformer architecture with depth-sampled 3D geometric representations. During training, natural language instructions are conditionally integrated to enhance understanding. The system optimizes for long-horizon accuracy and instruction alignment, employing AdamW optimizer with cosine annealing schedule.
Key Results
- On doScenes validation, the method achieved A@1s=0.145, A@2s=0.316, A@3s=0.613, reducing mADE from 0.372 to 0.358, outperforming baseline. On the official leaderboard, it ranked first with ADE=2.0913 and FDE=5.5946, significantly better than previous methods, demonstrating robustness and generalization.
Significance
This work advances multi-view perception by effectively reducing irrelevant feature interference, crucial for long-term trajectory prediction in autonomous driving. It enhances the model's ability to interpret natural language instructions and local visual cues, addressing key challenges in scene understanding and decision-making. The approach bridges vision and language modalities, paving the way for safer, more reliable autonomous systems capable of complex human-vehicle interactions.
Technical Contribution
The core innovation is the divided-view perception module, inspired by DVPE, which partitions the global 3D perception space into local views, reducing cross-view interference. Combined with Transformer-based feature aggregation and depth sampling, it improves local spatial reasoning. The conditional language integration and long-horizon training further distinguish this work from prior global interaction models, providing a new framework for instruction-conditioned trajectory prediction.
Novelty
This is the first application of divided-view perception in autonomous driving for instruction-conditioned trajectory prediction, addressing the limitations of global multi-view attention. It innovatively combines depth-sampled 3D geometric features with local spatial partitioning, significantly enhancing long-term prediction accuracy and interpretability, setting a new benchmark in multi-modal perception.
Limitations
- The model's performance degrades in scenarios with severe occlusion or rapid viewpoint changes, indicating a need for more robust occlusion handling. Training requires extensive multi-view data and computational resources, limiting real-time deployment. Its natural language understanding is limited to predefined instruction types, requiring further integration with advanced NLP modules.
Future Work
Future efforts include developing adaptive multi-view partitioning techniques, integrating reinforcement learning to optimize trajectory planning, and improving natural language understanding for more diverse instructions. Additionally, efforts will focus on reducing computational costs and deploying models in real-time systems, aiming for broader industrial adoption.
AI Executive Summary
Autonomous driving relies heavily on accurate trajectory prediction, yet traditional models struggle with long-term planning due to interference from irrelevant multi-view features. This paper introduces DVDrive, a novel approach that leverages a divided-view perception module inspired by DVPE, integrated within the OmniDrive framework. By partitioning the multi-view feature space into local virtual views based on 3D spatial information, the model reduces cross-view interference, enabling more precise local reasoning. Transformer-based feature aggregation combined with depth-sampled geometric representations enhances the model's understanding of complex scenes. Extensive experiments on nuScenes and doScenes datasets demonstrate that DVDrive outperforms existing methods, achieving a top ranking on the doScenes leaderboard with an ADE of 2.0913. This work significantly improves long-range trajectory prediction accuracy and instruction understanding, marking a step forward in vision-language autonomous driving systems. Looking ahead, further optimization and real-time deployment efforts will expand its practical impact, making autonomous vehicles safer and more reliable in diverse environments.
Deep Analysis
Background
自动驾驶技术经历了从激光雷达到多视角摄像头的感知演变，代表性工作如BEVFormer和Detr3D在多视角3D感知方面取得突破，但在长距离轨迹预测中仍受全局交互干扰限制。近年来，视觉-语言融合成为研究热点，旨在提升环境理解和人机交互能力。尽管如此，现有模型多忽视多视角特征的空间划分，导致信息冗余和干扰，影响长远轨迹的预测精度。随着自动驾驶场景复杂度提升，如何有效利用多视角信息，减少干扰，成为亟待解决的问题。
Core Problem
核心问题在于全局多视角交互引入大量无关信息，影响局部空间推理，尤其在长距离轨迹预测中表现不佳。传统模型难以有效利用局部空间信息，导致预测误差大、鲁棒性差。此外，模型对自然语言指令的理解能力有限，难以实现精准导航意图。这些问题限制了自动驾驶系统在复杂环境中的表现，亟需引入更细粒度的空间划分和多模态融合机制。
Innovation
引入DVPE风格的划分视角感知机制，将多视角空间划分为局部虚拟视图，减少跨视角干扰。结合深度几何采样，增强空间特征表达。利用Transformer实现局部空间的高效交互，提升长远轨迹预测精度。模型还融合自然语言指令，条件化训练增强理解能力。这些创新显著区别于传统全局交互方法，为多模态自主感知提供新思路。
Methodology
- �� 输入多视角图像特征，投影到统一Transformer空间。• 采样深度点，构建3D几何表示。• 将空间划分为局部视图，定义查询和特征集合。• 转换局部坐标系，生成位置编码。• 在每个局部视图内进行可见性感知的交叉注意力，聚合空间相关信息。• 融合历史场景信息，增强长距离预测能力。• 条件化自然语言指令，优化模型理解与执行。• 训练采用AdamW，结合指令增强监督，提升指令对齐。
Experiments
在nuScenes和doScenes数据集上，模型训练采用12个未来轨迹点，预测6秒轨迹。使用A@1s、A@2s、A@3s和mADE指标评估，训练细节包括学习率1e-4、余弦退火、批次2、8GPU。对比基线模型，提出方法在所有指标上均优，mADE从0.372降至0.358，验证了划分视角机制的有效性。模型在官方测试中排名第一，显著优于传统全局交互模型。
Results
模型在验证集上A@1s、A@2s、A@3s指标分别达0.145、0.316、0.613，优于基线。测试排名中，ADE为2.0913，FDE为5.5946，远低于历史和指令基线，验证了长远预测的准确性。引入划分视角机制后，模型在复杂场景中表现出更强的局部感知能力，有效减少干扰，提升指令与视觉证据的对齐。
Applications
该技术可应用于自动驾驶中的路径规划、场景理解和人机交互。依赖多视角摄像头和自然语言指令，适合未来智能驾驶辅助系统。模型提升了在复杂交通环境中的鲁棒性和预测精度，有助于实现更安全、更智能的自动驾驶。
Limitations & Outlook
模型在极端遮挡或视角剧烈变化场景下仍存在感知盲区，需进一步增强鲁棒性。训练成本高，依赖大量多视角数据，部署时对计算资源要求大。对复杂指令的理解能力有限，未来需结合更强的自然语言处理技术。
Plain Language Accessible to non-experts
想象你在一个繁忙的工厂里工作，工厂里有很多不同的区域和机器。每个区域都可以看作一个视角，工人需要根据指示完成任务。传统的方法就像让工人同时盯着所有区域，信息太多，容易迷失重点。而DVDrive的方法是把工厂划分成几个小区域，工人只关注自己负责的区域，减少干扰，更快找到目标。这样一来，无论工厂多大、多复杂，工人都能更准确地完成任务。这就像模型把多视角信息划分成局部视图，只关注与任务相关的部分，提升效率和准确性。
ELI14 Explained like you're 14
你知道在学校里，有很多教室和老师，每个老师负责不同的科目。有时候，老师会给你一些指示，比如“去操场跑步”或者“到图书馆找书”。如果你要完成一项任务，比如去操场跑步，你需要知道自己在哪个教室，怎么走，路上会遇到什么。传统的方法就像让你同时看所有教室的地图，信息太多，容易迷路。而新方法就像把学校划成几个区域，你只关注自己要去的区域，记住路线，这样就能更快找到目的地。这个想法用在自动驾驶中，就是把多视角的视觉信息划分成局部区域，让汽车更聪明、更快地理解环境，安全行驶。
Abstract
Instruction-conditioned trajectory prediction is an emerging problem in autonomous driving, where a model predicts the future ego trajectory not only from visual scene context and historical motion, but also from a natural-language maneuver instruction. This paper presents our submission to the doScenes Instructed Driving Challenge, built upon OmniDrive, a vision-language-action driving agent with 3D perception, reasoning, and planning capabilities. We adapt OmniDrive to the doScenes setting by training it on instruction-annotated nuScenes scenes and generating a 6-second ego trajectory represented by 12 future waypoints. To improve multi-view visual grounding, we further introduce a DVPE-style divided-view perception module into the OmniDrive perception head. Instead of attending globally to all camera features, the proposed module groups query features and image tokens into divided local view spaces and performs visibility-aware cross-attention within each view. This design reduces irrelevant cross-view interference and helps the model better align language instructions with local driving-relevant visual evidence. The code is publicly available at: https://github.com/feel12348/doscenes-omnidrive.