VLMs exposed: 'Multilingual' models fail script consistency test

A new paper from researchers introduces PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art Vision-Language Models (VLMs), they expose a significant Script Gap. Accuracy deltas between scripts reached 16% on identical visual tasks, and the proposed Script Consistency Rate (SCR) was as low as 24.8% for some models. Crucially, adding visual input boosted absolute performance uniformly but did not reduce the orthographic gap — models that scored high in one script still failed the same task in another.

The findings challenge the assumption that multilingual VLMs handle multi-script languages effectively. Cross-script in-context transfer was highly brittle, suggesting script-locked knowledge representations rather than true multi-script understanding. The authors call for SCR as a mandatory metric for equitable AI access, citing the billions of users of multi-script languages (e.g., Punjabi, Hindi, Urdu, Arabic). The study highlights that visual grounding alone is insufficient; models must be systematically tested for script consistency to avoid perpetuating bias against non-Roman or mixed-script communities.