Olmo2 and Pythia LLMs reveal late, fragile theory of mind

A new paper from researchers Rivière, Jones, and Trott takes a developmental perspective on LLM mentalizing abilities. Studying the Olmo2 and Pythia model suites across their training checkpoints, the authors measure performance on false belief tasks (FBT) and situation modeling—the ability to report basic factual properties of a described scene. They find that above-chance FBT performance emerges relatively late in pretraining, and is contingent on both model size and sufficient training volume. Among the conditions tested, post-training interventions like supervised fine-tuning (SFT) and direct preference optimization (DPO) yield the largest improvements, particularly in the most diagnostic condition: implicit false belief.

Yet FBT performance proves surprisingly fragile. Consistent with prior work, the use of non-factive verbs such as 'thinks' inflates false belief attributions even in true belief scenarios—a clear failure of robust mentalizing. Meanwhile, situation modeling accuracy generally precedes and exceeds FBT accuracy, but the representations are incoherent in key respects. For instance, Olmo2 13b’s judgments about an antagonist agent’s knowledge (who always knows the item’s true location) are consistently influenced by the target agent’s knowledge state and by non-factive verbs. The authors argue these results highlight the value of developmental and stress-testing approaches for evaluating LLM capabilities, showing that larger, sufficiently trained models build partially coherent situation models but with surprising blind spots.