VLADriveBench reveals when AI reasoning in self-driving is just talk
New benchmark finds ORION’s chain-of-thought looks good but doesn’t drive decisions—Alpamayo v1.5’s does.
A new paper from Thach Nguyen and colleagues at an undisclosed institution introduces VLADriveBench, a framework designed to evaluate whether the chain-of-thought (CoT) reasoning produced by vision-language-action (VLA) models for autonomous driving is actually causally connected to their driving actions. Current benchmarks only assess trajectory quality, ignoring whether the CoT is relevant, consistent, or causally linked to the action. VLADriveBench combines observational metrics—mentioning, hallucination, contradiction, and action alignment—with a novel CoT intervention protocol that tests causality by altering the reasoning and observing if the action changes.
Applying the framework to three models across two architectures, the results reveal a stark divergence. ORION achieved the highest observational scores for CoT-action alignment, yet its CoT turned out to be epiphenomenal—it looks correct but does not influence the driving decision. In contrast, Alpamayo v1.5 scored lower on observational metrics but its CoT was strongly causal: changing the reasoning actually changed the trajectory. The researchers also found that visual salience acts as a gate, determining how much the CoT influences the final action. This work underscores that high-quality reasoning on paper does not guarantee real decision-making and points to the need for deeper causal evaluation in autonomous driving AI.
- VLADriveBench uses both observational metrics and a causal intervention protocol to evaluate CoT-action relationships in VLA models.
- ORION scored highest on observational alignment but its CoT was epiphenomenal, while Alpamayo v1.5's CoT was strongly causal despite lower alignment scores.
- Visual salience gates the influence of CoT on driving actions, suggesting that what the model 'sees' matters more than its reasoning.
Why It Matters
Shows that fluent reasoning in autonomous driving AI can be misleading—real safety requires causal, not just plausible, decision-making.