VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving
VLADriveBench combines observational metrics and causal intervention to evaluate CoT–action causality in VLA autonomous driving models.
Key Findings
Methodology
VLADriveBench integrates multiple metrics—mentioning, hallucination, contradiction, action alignment—with a causal intervention protocol. It evaluates three models (ORION, Alpamayo R1, V1.5) in CARLA, using large multimodal language models (GPT-5.4, Claude Opus 4.6, Gemini Pro 2.5) for annotation. Observation metrics assess reasoning quality and correlation, while intervention tests causality by replacing reasoning segments. The framework is architecture-agnostic, enabling comprehensive analysis of CoT influence on driving actions.
Key Results
- ORION scores highest on observational alignment but its CoT is epiphenomenal, with limited causal influence on trajectories. Alpamayo v1.5, despite lower alignment scores, exhibits strong causal CoT, with visual salience gating influence. Intervention experiments show ORION’s CoT affects trajectory minimally, mainly through correlation, whereas Alpamayo v1.5’s CoT demonstrates clear causal effects, guiding safe behaviors.
Significance
This work advances evaluation beyond trajectory metrics, providing a multi-dimensional framework to understand the causal role of reasoning in autonomous driving. It addresses the critical need for trustworthy, interpretable models, especially in safety-critical scenarios. By combining observational and causal analyses, it offers insights into the true influence of CoT, guiding future model design and validation for safer deployment.
Technical Contribution
The paper introduces VLADriveBench, a modular, architecture-independent evaluation framework that merges quality metrics with causal intervention. It leverages large language models for automated annotation, implements self-splice validation, and conducts multi-scenario experiments. This approach enables precise diagnosis of whether CoT genuinely influences actions, setting a new standard for interpretability and safety validation in VLA models.
Novelty
This is the first comprehensive framework combining observational metrics with causal intervention to evaluate CoT–action relationships in autonomous driving models. Unlike prior methods focusing solely on trajectory accuracy, VLADriveBench provides a multi-faceted, causal perspective, revealing discrepancies between apparent alignment and true influence, thus offering a novel diagnostic paradigm.
Limitations
- The evaluation in CARLA environment may not fully reflect real-world complexities, especially since models like Alpamayo are trained on proprietary data and tested OOD, possibly inflating effect sizes. Real-world validation remains necessary.
- 干预机制在某些场景下可能受限于模型架构，难以捕捉深层因果关系。未来应结合多模态和因果推理技术提升准确性。
- 视觉显著性阈值受障碍物大小、距离和对比度影响，尚未全面量化。模型在不同环境中的表现仍需验证。
Future Work
未来将结合真实道路数据，优化因果推理机制，提升模型在复杂环境中的表现。计划开发实时监测与修正系统，确保推理的因果性和安全性。推动跨模型、跨任务的标准化评估体系，促进可信自动驾驶的发展。
AI Executive Summary
Autonomous driving models that integrate vision, language, and action—collectively called VLA models—are increasingly capable of generating not only trajectories but also chain-of-thought (CoT) reasoning. These models promise enhanced interpretability and generalization, but current evaluation methods fall short. Traditional benchmarks focus solely on trajectory accuracy metrics like L2 error or collision rate, neglecting whether the reasoning content genuinely influences the driving decisions. This gap hampers understanding of the models’ decision-making processes and raises safety concerns.
To address this, VLADriveBench introduces a comprehensive evaluation framework that combines observational metrics—such as mention accuracy, hallucination detection, contradiction identification, and action alignment—with causal intervention protocols. The observational metrics assess the quality and relevance of the reasoning, while the intervention tests whether replacing parts of the CoT alters the driving action, thus establishing causality. This dual approach enables a nuanced understanding of whether the reasoning truly guides behavior.
Applying VLADriveBench to three models—ORION, Alpamayo R1, and V1.5—in the CARLA simulator revealed surprising divergences. ORION scored highest on observational alignment but its CoT was found to be epiphenomenal, meaning it correlated with actions without causally influencing them. Conversely, Alpamayo v1.5, despite lower alignment scores, demonstrated strong causal relationships, with visual salience gating the extent of CoT influence. These findings underscore the importance of combining multiple evaluation dimensions to accurately interpret model reasoning.
This work significantly impacts the development of trustworthy autonomous systems. By providing a modular, architecture-agnostic toolkit, VLADriveBench enables researchers and developers to systematically diagnose the true influence of reasoning in their models. It paves the way for safer, more interpretable autonomous driving solutions, especially critical in safety-sensitive environments. Future efforts will focus on extending this framework to real-world data, refining causal inference methods, and establishing industry standards for reasoning validation, ultimately fostering greater trust and reliability in autonomous vehicles.
Deep Dive
Abstract
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.