Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Driving

A team from the University of Colorado Boulder has published a critical paper addressing a major flaw in how AI for self-driving cars is evaluated. The research, 'Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Driving,' reveals that Vision Language Models (VLMs) can exploit hidden linguistic cues in synthetically generated Multiple-Choice Question Answering (MCQA) benchmarks, achieving high accuracy without actually understanding visual scenes. The authors—Sutej Kulgod, Sean Ye, Sanchit Tanwar, and Christoffer Heckman—demonstrated that a VLM fine-tuned on such biased data could match human-validated benchmark performance even when deprived of visual input, a clear sign of 'cheating' via textual shortcuts.

The core of their proposed method involves two key technical interventions. First, they systematically decouple the correct answer in a question from any linguistic artifacts or patterns that could serve as a giveaway. Second, they employ a curriculum learning strategy that gradually increases the difficulty of identifying and ignoring these biases. The results are striking: their method reduced what they term 'blind accuracy'—performance attributable solely to text patterns—from +66.9% above random chance down to a mere +2.9%. This drastic reduction effectively eliminates the vast majority of exploitable textual shortcuts.

This work matters because MCQA benchmarks are a standard tool for measuring VLM performance in critical tasks like autonomous driving. If models are scoring well by 'reading the test' rather than 'seeing the road,' safety evaluations are fundamentally flawed. The new method forces models to rely on genuine visual grounding, ensuring that reported accuracy metrics accurately reflect a model's perceptual understanding of complex driving scenarios. This is a vital step toward creating more robust, reliable, and truly vision-dependent AI systems for real-world autonomous vehicles.