Study: VLMs don't uniformly beat LLMs in mimicking human reading

A team of researchers led by Jinzhou Wu compared large language models (LLMs) and vision-language models (VLMs) to test whether multimodal pretraining makes text representations more human-like during natural reading. They used tightly matched model pairs under a strictly text-only setting to isolate the effect of multimodal training history from online visual input. Human alignment was measured using whole-cortex fMRI responses and synchronized eye-tracking saccades from a natural reading dataset.

Results show that multimodal pretraining does not confer a uniform, global advantage. VLMs only outperformed LLMs when sentences contained stronger visual semantic content, and this effect was visible in both brain activity and eye movement patterns. The findings challenge the assumption that adding vision to language models automatically improves their alignment with human cognition for reading tasks.