EEG Study Reveals LLMs Don't Mimic Human Brain's Word Prediction

A new study published on arXiv examines whether advanced language models (LMs) process language similarly to humans by analyzing EEG brain signals during reading. The researchers — from Dublin City University and Vietnam National University — generated two types of regressors for both humans and LMs: top-1 prediction accuracy and surprisal (a measure of word unexpectedness). These were used to predict event-related potentials (ERPs) recorded from EEG, which reflect cognitive processing stages. The goal was to see if higher prediction accuracy in LMs correlates with human-like neural responses.

The results show a clear distinction: only surprisal correlates with language-processing ERPs, especially for open-class words (nouns, verbs, adjectives). Top-1 prediction accuracy, even in state-of-the-art models, does not align with brain signals. This challenges the assumption that scaling up parameter counts and compute budgets consistently improves convergence with human linguistic processing. The findings suggest that current LLM training objectives may not capture the nuanced cognitive mechanisms behind human reading, raising questions about how to better align AI with neural processing.