Grounding Driving VLA via Inverse Kinematics
Transforming driving VLA into an inverse kinematics framework with future visual prediction and diffusion-based IK network, achieving performance comparable to much larger models.
Key Findings
Methodology
This work reformulates the driving VLA task as an inverse kinematics problem, introducing a future visual state prediction objective to provide dense visual supervision. A cross-attention conditional diffusion model-based IK network is designed to decode trajectories solely from current and predicted future visual states, avoiding reliance on ego status or textual shortcuts. The training jointly optimizes future visual state prediction and trajectory reconstruction, encouraging the model to learn causal visual features. This approach effectively addresses the shortcut learning issue prevalent in end-to-end models, which tend to ignore visual tokens due to the lack of explicit supervision and architectural biases.
Key Results
- On NAVSIM-v2 and nuScenes benchmarks, our 0.5B model achieves performance on par with 7B-8B VLAs, with PDMS improving by 19.0% and EPDMS by 20.4%. Collision rates are significantly reduced, demonstrating enhanced safety and robustness. Visual attribution analyses confirm the model’s reliance on visual features, especially in dynamic scenarios like turning.
- Counterfactual obstacle stitching experiments show the model’s trajectory adapts to visual changes, indicating effective visual grounding. Ablation studies reveal that future visual state prediction and the IK network are critical for performance gains.
- The model’s ability to exploit visual cues leads to superior safety metrics and generalization, surpassing larger models in complex driving environments.
Significance
This research addresses a fundamental flaw in current end-to-end driving models: the tendency to ignore visual information and rely on shortcuts. By systematically integrating future visual states and a dedicated inverse kinematics module, the approach restores visual grounding, improves trajectory accuracy, and enhances safety. It offers a new paradigm for designing scalable, interpretable, and safer autonomous driving systems, bridging the gap between perception and planning. The findings suggest that structural task reformulation can outperform mere scaling of model size, providing valuable insights for future research and industry deployment.
Technical Contribution
The core technical innovation lies in reformulating trajectory prediction as an inverse kinematics problem, with a dedicated diffusion-based IK network conditioned only on current and future visual states. The model jointly trains a dense future visual prediction task, which provides continuous visual supervision, and employs a cross-attention diffusion model to decode trajectories, effectively preventing shortcut learning. The architecture’s separation of trajectory decoding from ego states and commands enhances visual feature exploitation. Theoretical analysis via bias-variance decomposition confirms the reduction of shortcut bias, and extensive experiments validate the approach’s effectiveness in real-world benchmarks.
Novelty
This work is the first to explicitly model driving trajectory prediction as an inverse kinematics problem, leveraging future visual states as boundary conditions. Unlike prior end-to-end models that rely on direct regression from ego states and commands, this approach emphasizes causal visual features, employing a novel diffusion-based IK network. The integration of dense future visual prediction as supervision and the architectural separation of trajectory decoding constitute a significant departure from existing methods, enabling more robust and interpretable visual grounding in dynamic driving scenarios.
Limitations
- The approach depends heavily on the accuracy of future visual state prediction; errors in visual encoding or prediction can degrade trajectory quality, especially in highly complex or occluded environments.
- The diffusion-based IK network introduces additional computational overhead, which may challenge real-time deployment without further optimization.
- Current evaluations are primarily on simulation and limited real-world datasets; broader testing in diverse, real-world scenarios is necessary to validate generalization.
Future Work
Future directions include integrating multi-modal sensory data (LiDAR, radar) to improve robustness, optimizing the diffusion IK network for real-time inference, and exploring reinforcement learning to adaptively improve visual prediction accuracy. Extending the framework to handle longer horizon planning and more complex scenarios, such as multi-agent interactions, will further enhance its practical deployment in autonomous vehicles.
AI Executive Summary
Autonomous driving has seen rapid advancements through vision-language-action models (VLA), which aim to unify perception, prediction, and planning into an end-to-end framework. Despite their promise, existing VLAs often suffer from a critical flaw: they tend to ignore visual information, relying instead on ego states and textual commands. This phenomenon, known as blind planning, limits their ability to respond adaptively to dynamic environments, especially in complex scenarios like obstacle avoidance and turning.
This paper introduces a novel approach that redefines the driving VLA task as an inverse kinematics problem. The key idea is to incorporate a future visual state prediction task, which provides dense supervision and encourages the model to ground its trajectory in visual features. Additionally, a dedicated inverse kinematics network based on a cross-attention conditional diffusion model is designed to decode trajectories solely from current and predicted future visual states, effectively suppressing shortcuts through ego states and commands.
The methodology involves extracting visual tokens from multi-camera inputs using a DINOv3 ViT encoder, training the model to predict future BEV visual states, and jointly optimizing this prediction with trajectory reconstruction. The IK network then decodes trajectories conditioned only on visual states, ensuring causal visual grounding. Extensive experiments on NAVSIM-v2 and nuScenes benchmarks demonstrate that the proposed model, with only 0.5B parameters, surpasses larger models (7B-8B) in key metrics such as PDMS, EPDMS, and collision safety, achieving performance comparable to or better than state-of-the-art large-scale VLAs.
Analysis confirms that the model’s improved performance stems from its recovered ability to exploit visual features, especially in dynamic scenarios like turning. Ablation studies highlight the importance of future visual prediction and the IK network, while counterfactual obstacle stitching validates the causal influence of visual cues. These results suggest that structural task reformulation, emphasizing causal visual grounding, can significantly enhance the safety and robustness of autonomous driving systems.
Looking ahead, future work will focus on multi-modal data integration, real-time inference optimization, and broader real-world validation. This approach offers a promising pathway toward safer, more interpretable, and scalable autonomous driving solutions, fundamentally shifting the paradigm from shortcut-dependent models to causally grounded perception and planning.
Deep Analysis
Background
自动驾驶技术经历了从传统模块化感知、预测、规划到端到端深度学习模型的演变。早期方法依赖感知模块提取环境信息，结合路径规划实现自主驾驶，但受限于信息传递延迟和误差累积。近年来，诸如DriveVLA、ARTEMIS等模型尝试融合视觉、文本信息，提升整体性能。然而，这些模型普遍存在对视觉信息利用不足的问题，表现为在复杂动态场景中缺乏有效的因果推理能力。研究发现，模型倾向于依赖ego状态和文本指令，忽略视觉特征的因果关系，导致盲目规划和安全风险。尽管已有引入未来视觉状态预测的尝试，但缺乏系统性设计，捷径学习问题依然严重，限制了模型的泛化和鲁棒性。
Core Problem
核心问题在于现有驾驶VLA模型未能充分利用视觉信息，导致在复杂动态环境中表现不佳。模型倾向于捷径学习，只依赖ego状态和文本指令进行轨迹预测，忽略视觉特征的因果关系。这不仅影响路径合理性，也增加安全风险。问题根源在于任务设计的结构性缺陷：只利用当前视觉状态作为输入，未引入未来状态作为边界条件，限制了模型的因果推理能力。这一缺陷使模型难以在复杂场景中做出合理决策，亟需结构性改进。
Innovation
本研究的创新点包括：1）将轨迹预测任务转化为逆运动学问题，强调未来视觉状态的预测作为边界条件，增强因果关系；2）设计基于跨注意力条件扩散模型的逆运动学（IK）网络，仅依赖当前与未来视觉状态，抑制捷径路径；3）引入密集的未来视觉状态预测任务，提供丰富视觉监督，促使模型学习视觉特征的因果关系。这些创新突破了传统端到端模型的局限，有效解决捷径学习问题，提升视觉引导的轨迹规划能力，为自动驾驶提供更可靠的决策基础。
Methodology
- �� 视觉特征提取：利用DINOv3 ViT-S/16编码多角度摄像头图像，得到视觉token。• 未来状态预测：模型通过LLM预测未来N秒的BEV视觉状态，形成密集视觉监督。• 逆运动学网络：设计基于跨注意力条件扩散模型的IK网络，仅依赖当前和预测的未来视觉状态，解码轨迹。• 损失函数：同时优化未来状态预测和轨迹重建，确保视觉特征的因果关系。• 训练策略：联合训练模型，利用反事实拼接验证视觉依赖，增强模型鲁棒性。• 预测流程：输入当前视觉状态，预测未来视觉状态，再由IK网络解码轨迹，避免捷径学习。
Experiments
模型在NAVSIM-v2和nuScenes两个基准上验证。训练采用两阶段策略，调优超参数如学习率、批次大小。对比基线包括传统端到端模型和大规模VLAs。指标涵盖PDMS、EPDMS、Collision Rate等，特别关注模型对视觉信息的利用。通过GradCAM和反事实拼接验证模型对视觉特征的依赖性。进行消融实验，验证未来状态预测和IK网络的贡献。结果显示，模型在性能和安全指标上均优于对比方法，验证了设计的有效性。
Results
在NAVSIM-v2中，0.5B模型的PDMS达90.6，较OpenDriveVLA提升19.0%，在nuScenes中，ADE降至0.06，Collision Rate保持在0.09，均优于更大规模模型。反事实拼接实验中，模型对障碍物的反应明显增强，表现出对视觉特征的依赖。消融实验表明，未来视觉状态预测和逆运动学网络的加入是性能提升的关键因素。这些结果验证了模型在动态场景中的鲁棒性和视觉引导能力。
Applications
该模型适用于自动驾驶中的路径规划与场景理解，尤其在复杂动态环境中表现优异。结合感知模块实现端到端安全驾驶，提升系统的可靠性。未来可扩展到多模态融合、强化学习等场景，推动智能驾驶的商业化应用。
Limitations & Outlook
模型高度依赖未来视觉状态的预测准确性，视觉编码或预测误差可能影响轨迹质量，尤其在极端复杂或遮挡环境中表现不足。引入逆运动学网络增加计算复杂度，实时性仍需优化。当前验证主要在模拟和有限真实场景中，泛化能力有待提升。未来需解决多模态融合、长时序预测等挑战。
Plain Language Accessible to non-experts
想象你在开车，但你不仅要看眼前的路，还要提前知道未来几秒会发生什么。传统的自动驾驶系统就像只看眼前的路，遇到障碍时反应慢或不够聪明。而本文的方法就像给车子装上了“预知未来”的眼睛，它可以提前预测未来几秒的路况，然后根据这个预测提前做出反应。这样一来，车子在转弯、避障时就像有了“预知未来”的能力，能更快更安全地做出决策。它用一种特别的技术，把未来的视觉信息预测出来，然后用一个聪明的“机器人”帮忙规划路线，确保车子既快又稳。这种方法让自动驾驶变得更像一个有预知能力的“未来预警系统”，大大提高了安全性和效率。
ELI14 Explained like you're 14
嘿，你知道吗？开车的时候，不只是看眼前的路，还要猜猜未来几秒会发生什么。以前的自动驾驶就像只盯着眼前的路，遇到突然出现的障碍就反应慢。现在，这个新方法就像给车子装了“未来预知眼睛”，它可以提前告诉车子未来几秒会遇到什么，然后帮它提前准备。它用一种特别的“预言”技术，把未来的路况预测出来，然后用一个聪明的“机器人”帮忙规划路线，确保车子既快又安全。这样一来，车子在转弯、避障时就像有了“预知未来”的能力，不会突然撞到东西。这个技术让自动驾驶变得更聪明、更安全，就像你在玩游戏时用作弊码提前知道下一关的秘密一样酷！未来，这样的技术可以让我们的车子像超级英雄一样，提前预防所有危险，开得又快又稳。
Abstract
Existing Driving VLAs predict trajectories while largely ignoring their visual tokens -- a phenomenon we trace not to insufficient training but to a structurally ill-posed task formulation. We show that trajectory recovery, when viewed through the lens of inverse kinematics, requires both a current and a future visual state as boundary conditions; existing VLAs supply only the former, which encourages the model to shortcut through ego status and text commands alone. To address this, we re-design Driving VLA in the style of an inverse kinematics solver. First, a next visual state prediction objective that requires the LLM to predict the future visual scene provides dense visual supervision and suppresses shortcut paths. Second, a separate Inverse Kinematics Network (a cross-attention-based conditional diffusion model) that takes only the current and future visual states as input is designed to suppress reliance on ego status and textual shortcuts during trajectory decoding. With this simple prescription alone, our 0.5B-scale model recovers visual grounding and reaches trajectory planning performance comparable to 7B--8B VLAs more than an order of magnitude larger, on both the closed-loop NAVSIM-v2 and the nuScenes benchmarks. Extensive analysis further shows that this improvement stems from a recovered ability to exploit visual features, with the effect being most pronounced in dynamic driving situations such as turning.