Entity tracking emerges in 410M-parameter LMs, beating humans

Language comprehension requires more than pattern matching: readers must track entities — characters, objects, locations — across a narrative, even when their state changes implicitly. A new arXiv paper from researchers Karolina Drożdż and Micha Heilbron investigates whether language models (LMs) perform this 'entity tracking' in a human-like way, and at what scale the capability emerges. Prior evaluations relied on artificial tasks that don't reflect natural language, and rarely compared models against humans. This study used naturalistic narratives with varying levels of complexity and had 48 human participants complete tracking probes, while a range of LM sizes (from sub-billion to contemporary large models) ran the same task.

The findings are striking: entity tracking already appears at 410 million parameters — an order of magnitude smaller than the multi-billion-parameter, code-specialised models highlighted by earlier work. The capability improves with scale, and today's state-of-the-art models far surpass human performance on these naturalistic narratives. Interestingly, human performance degrades specifically with narrative complexity — not with length — suggesting limits in working memory or discourse integration. The authors argue this demonstrates that a core building block of language understanding emerges at model scales far smaller than previously assumed, with implications for efficiency: smaller LMs might handle context-tracking tasks without needing massive architectures. Beyond efficiency, the findings open a new window into comparing machine and human comprehension: by using naturalistic stimuli instead of synthetic probes, the benchmark offers a more faithful measure of everyday language understanding, and could help guide future model design toward human-like robustness in long-form reasoning.