Probing for Reading Times
New study shows language models encode human-like cognitive signals, with early layers outperforming traditional metrics.
A team of researchers from ETH Zurich and the University of Cambridge, led by Eleftheria Tsipidi, published a groundbreaking paper titled "Probing for Reading Times" at ACL 2026. The study investigates whether the internal representations of large language models (LLMs) like GPT and Llama encode not just linguistic information but also cognitive signals related to human language processing. Using regularized linear regression, they probed model layers against human eye-tracking data from two corpora spanning five languages: English, Greek, Hebrew, Russian, and Turkish.
The key finding is a functional alignment between model architecture and human cognition. The representations from the early layers of these models consistently outperformed traditional scalar predictors like surprisal and information value when predicting early-pass reading measures, such as first fixation duration and gaze duration. This suggests that low-level, structural representations within AI models capture signatures of initial human word recognition. However, for late-pass measures like total reading time, the simpler scalar surprisal metric remained superior, indicating different cognitive processes are at play.
This research provides a new lens for evaluating AI models, moving beyond pure task performance to understanding how they process information relative to humans. The discovery that early layers are most predictive of early reading stages could inform the development of more cognitively plausible models and improve techniques for interpreting model behavior. The performance also varied significantly by language and specific eye-tracking measure, highlighting the complexity of cross-linguistic cognitive modeling.
- Early model layers (like those in GPT/Llama) beat surprisal at predicting early human reading measures (first fixation, gaze duration) across 5 languages.
- The study found a functional alignment: early layers map to early reading stages, while late-pass reading (total time) is better predicted by simpler surprisal.
- Performance gains were achieved by combining surprisal with early-layer representations, and results varied strongly by both language and specific eye-tracking metric.
Why It Matters
Provides a new cognitive benchmark for AI, linking model internals to human processing, which can guide development of more interpretable and human-like systems.