LLM Theory of Mind gains don't translate to real interactions

A new study by Nanxu Gong and collaborators from multiple institutions challenges the assumption that improving Theory of Mind (ToM) in large language models (LLMs) via static benchmarks translates to better human-AI interactions. The researchers argue that existing ToM tests—typically story-reading, third-person multiple-choice questions—fail to capture the first-person, dynamic, and open-ended nature of real conversations. To bridge this gap, they developed a novel interactive evaluation paradigm that shifts both perspective (from third-person to first-person) and metrics (from static accuracy to interaction quality).

Using four representative ToM enhancement techniques, the team tested LLMs across four real-world datasets and a user study covering goal-oriented tasks (coding, math) and experience-oriented tasks (counseling). Their key finding: static ToM improvements do not consistently predict better performance in dynamic interactions. In some cases, models with higher static ToM scores performed worse in interactive settings, suggesting that current benchmarks may reward superficial reasoning patterns. The paper calls for interaction-based assessments to build truly socially aware LLMs for human-AI symbiosis.