AI models GPT-4o, Claude, Gemini fail classic attention test at scale

Researchers led by Suketu Patel put top AI language models through the Stroop task, a classic psychological test of attention and impulse control. In the test, subjects must name the ink color of a printed color word (e.g., say “blue” for the word “red” printed in blue ink). Humans handle this easily even with long lists, but the AI models showed a dramatic breakdown as list length increased. GPT-4o achieved 91% accuracy on five-word lists, but that fell to 57% at ten words and just 15% at forty words. Claude 3.5 Sonnet maintained performance up to twenty words, then plunged to 24% at forty words. GPT-5, Claude Opus 4.1, and Gemini 2.5 followed similar patterns. When matching and mismatched words were mixed, accuracy for mismatched items sometimes dropped to nearly zero.

These results highlight a fundamental gap between machine and human attention. While LLMs excel at single-shot reasoning or short prompts, they lack the executive control to consistently follow a single instruction across longer sequences. Instead, they default to their dominant training pattern—reading words rather than naming colors. The study suggests that current AI systems cannot sustain goal-directed focus when cognitive load increases, a weakness that could affect real-world applications like long-form document analysis, multi-step reasoning, or any task requiring sustained attention to detail. The findings were published in PNAS Nexus.