PerceptDrive: Perception Prior World-Action Modeling with Adaptive Expert Routing for End-to-End Autonomous Driving
PerceptDrive uses adaptive expert routing to fuse perception priors, achieving 90.4 PDMS on NAVSIM v1 without test-time candidate search.
Key Findings
Methodology
PerceptDrive employs frozen perception providers (VGGT, V-JEPA, Wan) to extract geometric, semantic, and dynamic priors, combined with dense observation latents. During training, expert branches are anchored via retention objectives (Lret) to preserve source info, with scene-conditioned soft gating (gr) dynamically adjusting expert contributions. The core components include: • query banks compress perception features; • expert-specific retention targets prevent info collapse; • scene-conditioned gating modulates expert fusion; • soft-gate distillation during training ensures single-trajectory inference. The model integrates future latent conditions with flow-matching for trajectory generation.
Key Results
- On NAVSIM v1, PerceptDrive achieves 90.4 PDMS, surpassing DriveWAM (90.1) and DriveDreamer (89.2). On NAVSIM v2, EPDMS reaches 90.2, outperforming all baselines. The model operates without candidate scoring or reranking, demonstrating stability (std<0.05).
- Ablation studies confirm that prior retention and scene-conditioned routing provide complementary gains. The model effectively fuses geometric, semantic, and dynamic priors, adapting to scene variations.
- Inference is a single forward pass, avoiding test-time candidate search, which simplifies deployment and enhances real-time performance.
Significance
This work advances end-to-end autonomous driving by effectively integrating multi-source perception priors through scene-aware adaptive routing. Unlike static fusion or parameter-based experts, PerceptDrive dynamically modulates prior contributions, maintaining source diversity and improving robustness in complex environments. Its ability to generate trajectories directly, without candidate evaluation, simplifies the pipeline and reduces latency. The approach bridges the gap between perception knowledge and planning, offering a scalable, efficient solution for real-world deployment. It also opens avenues for further multi-source fusion and scene adaptation strategies, pushing autonomous systems toward safer, more reliable operation.
Technical Contribution
The paper introduces a novel framework combining frozen perception models with trainable expert-specific branches, anchored by retention objectives to preserve source information. Scene-conditioned gating (gr) predicts soft weights for expert fusion, enabling dynamic scene adaptation. The integration of future latent conditions with flow-matching-based trajectory prediction enhances continuity and robustness. The training employs soft-gate distillation, leveraging privileged sub-metrics, to ensure single-trajectory inference without test-time candidate scoring. This approach differs from prior static or parameter-based expert routing, providing a flexible, scene-aware fusion mechanism that significantly improves performance and simplifies inference.
Novelty
This is the first work to implement a scene-conditioned, adaptive expert routing mechanism for perception prior fusion in end-to-end autonomous driving. Unlike previous static fusion or scene-invariant expert models, PerceptDrive dynamically adjusts expert contributions based on scene context, maintaining source-specific information through retention objectives. Its integration of privileged supervision during training with a single forward inference path marks a significant innovation, enabling high performance without test-time candidate search. This approach fundamentally enhances the ability to leverage heterogeneous perception priors in real-time trajectory planning.
Limitations
- 依赖大量预训练感知模型，训练成本高，模型性能受感知模型偏差影响较大，可能在极端或未见场景中表现不足。
- 场景条件路由在复杂或未训练场景中可能不足以充分调节专家贡献，影响鲁棒性。
- 推理阶段未考虑多轨迹或候选路径，限制在多目标、多路径场景中的应用，未来需引入多轨迹预测机制。
Future Work
未来可结合多轨迹生成和多模态感知，提升复杂场景的适应性。探索更高效的感知模型压缩与融合策略，降低训练成本。引入强化学习或在线适应机制，增强模型在未见场景中的泛化能力。还可结合地图信息和激光雷达数据，丰富感知先验，推动自主驾驶系统的智能化和安全性提升。
AI Executive Summary
自主驾驶技术正迎来快速变革，然而多源感知信息的高效融合仍是瓶颈。传统方法多采用静态融合或参数调节，难以应对复杂多变的交通场景。本文提出的PerceptDrive框架，创新性地引入场景条件下的自适应专家路由机制，有效融合几何、语义和动态感知先验。通过在训练中利用软门控蒸馏和留存目标，模型能够在推理时实现单轨轨迹生成，无需候选排序或重排，极大简化流程。实验结果显示，在NAVSIM v1和v2数据集上，模型分别达到了90.4和90.2的PDMS指标，优于所有对比方法，验证了其优越性能。该方法不仅提升了轨迹预测的准确性，也增强了模型在复杂环境中的鲁棒性。未来，结合多轨迹预测和多模态感知，有望推动自主驾驶系统向更智能、更安全的方向发展。
Deep Analysis
Background
自主驾驶经历了从传统模块化到端到端深度学习的转变。早期依赖规则和手工设计，存在信息传递瓶颈。近年来，基于视觉语言模型（VLM）和世界模型（World Models）的方法不断涌现，如DriveVLA、Law、DriveDreamer等，提升了感知理解和路径规划能力。然而，如何融合多源感知信息，保持信息完整性并实现场景适应，仍是难点。静态融合方法无法动态调节专家贡献，导致信息丢失。多专家路由和动态调节机制逐渐成为研究热点，但在保持源信息多样性和推理效率方面仍有挑战。
Core Problem
核心问题在于如何在端到端规划中，有效融合几何、语义和动态先验，既保证信息的完整性，又能根据场景动态调节专家贡献。现有方法多采用静态加权或参数调节，难以实现场景依赖的动态调控，导致信息丢失或融合不充分。这限制了模型在复杂交通环境中的表现，影响安全性和鲁棒性。解决方案需设计一种机制，既能在训练中保持多源信息，又能在推理中实现场景适应。
Innovation
创新点包括：1）引入场景条件的软门控专家路由（scene-conditioned routing），实现动态调节几何、语义、动态先验的贡献；2）设计专家特定的留存目标（Lret），确保信息不崩溃；3）结合未来潜在条件和流匹配策略，提升轨迹连续性。不同于传统静态融合或参数调节，本方法在推理阶段实现了场景依赖的动态调节，显著提升了端到端路径规划的表现，特别是在复杂交通环境中。
Methodology
- �� 构建冻结的感知提供者：利用VGGT、V-JEPA和Wan模型提取几何、语义、动态先验，结合密集观察潜在特征。
- �� 设计专家分支：每个专家对应一种先验，利用留存目标（Lret）保持信息，利用查询银行压缩特征。
- �� 场景条件调节：通过场景表示（st）预测软门控（gr），动态调节专家贡献，融合专家条件。
- �� 训练过程：引入软门控蒸馏，将训练中优质轨迹草稿作为目标，确保推理时单轨输出。
- �� 轨迹生成：结合未来潜在条件和流匹配策略，利用动作预测头和流模型，输出轨迹。
Experiments
在NAVSIM v1和v2数据集上，模型采用离线感知模型和多教师蒸馏，训练目标包括轨迹拟合、门控蒸馏和未来潜在预测。评估指标为PDMS和EPDMS，模型对不同场景表现稳定。消融实验验证专家留存和场景条件调节的贡献，模型在复杂交通环境中表现优越，且无需候选排序，推理快速。
Results
在NAVSIM v1中，达90.4 PDMS，优于DriveWAM（90.1）和DriveDreamer（89.2）；NAVSIM v2中，EPDMS达90.2，优于所有对比方法。消融实验显示专家留存和场景调节互补提升性能。模型在极端场景（navhard）中也表现出色（EPDMS 34.5），验证其鲁棒性。
Applications
可应用于自动驾驶中的路径规划，尤其适合单摄像头场景，简化推理流程，提升实时性。未来可结合多轨迹预测、多模态感知，增强复杂环境中的适应能力，推动自动驾驶的安全性和效率。
Limitations & Outlook
模型依赖大量预训练的感知模型，训练成本高，感知偏差可能影响整体性能。在极端或未见场景中，场景条件路由可能不足以充分调节专家贡献，导致性能下降。推理中未考虑多轨迹生成或候选路径，可能在多目标或多路径场景中表现有限，未来需引入多轨迹预测机制。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有很多不同的工具和食材，比如刀、锅、蔬菜、调料。每个工具和食材都代表一种信息，比如刀代表切割技巧，蔬菜代表环境信息。现在，你需要根据不同的菜谱（场景）灵活使用这些工具和食材。传统的方法就像用同一种调料，不管菜谱怎么变都用一样的调料，效果不够好。这个新方法像厨师能根据菜谱自动调节用料比例，灵活搭配不同的工具，做出更美味的菜。它通过学习不同的厨艺技巧（专家），在每个菜谱（场景）中选择最合适的搭配，确保每次都能做出满意的菜肴。这就像在复杂的交通环境中，系统能根据不同的场景，灵活调节感知信息的利用，保证行车安全和效率。
ELI14 Explained like you're 14
想象你在玩一款超级复杂的游戏，你要控制一个角色穿越各种不同的场景，比如城市、森林或沙漠。每个场景都有不同的挑战，比如交通、动物或沙尘暴。以前的方法就像用同一种策略应对所有场景，不管遇到什么都用一样的技能，结果可能不太好。现在，这个新系统像你的游戏助手，能根据场景自动选择最合适的技能和工具，比如在城市用车、在森林用弓箭。它通过学习各种场景的特点，把不同的“技能”存储在不同的“专家”里，然后根据当前场景，智能调节用哪个“专家”的技能。这样，角色就能更聪明、更快地应对各种挑战，保证顺利完成任务。这就像在交通自动驾驶中，系统能根据不同的路况，灵活调节感知信息的利用，确保行车安全和顺畅。
Glossary
感知先验 (Perception Priors)
预训练模型提供几何、语义和动态信息，用于辅助决策。技术上指利用预训练模型提取的场景知识。
在本文中，感知先验作为专家分支的条件输入，用于轨迹预测。
专家路由 (Expert Routing)
根据场景动态调节不同专家条件的贡献，通过软门控实现信息融合。技术上是利用场景表示预测门控权重。
用于自适应融合几何、语义、动态先验。
留存目标 (Retention Objective)
保持每个专家分支的源信息完整，防止信息崩溃。通过重建目标实现。
确保专家分支能保持源信息的多样性。
流匹配 (Flow-Matching)
一种轨迹生成策略，通过匹配未来潜在的流式特征，连续预测路径。技术上结合潜在条件和动作。
用于轨迹生成，提升连续性和鲁棒性。
软门控蒸馏 (Soft-Gate Distillation)
在训练中利用优质轨迹草稿作为软目标，蒸馏门控信息，确保推理时单轨输出。
提升模型推理效率和性能。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂或未见过的场景中，保持专家路由的鲁棒性和适应性仍是挑战，未来需结合在线学习或多轨迹预测机制。
Applications
Immediate Applications
单摄像头路径规划
在自动驾驶中，利用单前视摄像头实现高精度轨迹预测，无需复杂候选排序，提升实时性和安全性。
Long-term Vision
多模态感知融合
结合激光雷达、高清地图等多源信息，增强感知先验的丰富性与鲁棒性，推动自动驾驶系统的智能化。
Abstract
Frozen perception foundation models encode rich geometric, semantic, and dynamic knowledge. Yet narrow conditioning interfaces may attenuate task-relevant cues, while static fusion cannot adjust expert contributions to each scene. We cast this challenge as the prior-to-plan transfer problem and introduce PerceptDrive, a perception prior world-action modeling framework with adaptive expert routing. PerceptDrive feeds teacher-distilled priors from a frozen, driving-adapted provider and dense observation latents from a frozen self-supervised video encoder into a trainable expert-routed world-action model. Expert-specific query branches process these signals, while a prior-retention objective anchors each branch to its prior. A router predicts soft gates from a shared scene representation and combines the expert conditions before trajectory generation. During training, privileged rule-based sub-metric estimates for branch-specific trajectory drafts provide soft-gate distillation targets. The predicted action-free future latent conditions a flow-matching actor. At inference, privileged components are absent; with one front-facing camera, PerceptDrive generates one trajectory per planning step without test-time scoring, reranking, or search. Experiments show that PerceptDrive achieves state-of-the-art performance with 90.4 PDMS on NAVSIM v1 and 90.2 EPDMS on NAVSIM v2, outperforming existing methods. Ablations confirm complementary gains from prior retention and scene-conditioned routing, alongside differential reliance on the three priors. These results demonstrate that preserving and adaptively routing perception priors improves direct planning without test-time candidate selection.