New Test Reveals When AI Outsmarts You — and When It Fails

When you play chess or negotiate a deal, you think about what the other person might do next. That's called strategic reasoning, and it's something new AI tools are supposed to get better at. But until now, most AI tests used classic games like poker or rock-paper-scissors that the AI may have already seen during training. That means it could just memorize the right answer instead of actually reasoning.

This paper from researchers Binchi Zhang and Atrisha Sarkar fixes that problem. They invented completely new games, so the AI couldn't cheat by remembering past answers. Then they tested four different AI models at ten levels of thinking depth — from "what will I do?" up to "what will they think I think they'll do?" The results were surprisingly clear.

AI systems did really well when asked to think out loud step-by-step, showing they can actually carry out deep chains of reasoning. But when they had to figure out an opponent's strategy just by watching how that opponent played, accuracy dropped sharply and unevenly across games. The researchers also found that mistakes came from choosing the wrong number of reasoning steps, not from making a bad move at a given step. Encouragingly, when the AI explicitly said what it imagined the opponent was thinking, its performance improved a lot.

The practical takeaway is that AI strategic ability is real but inconsistent. In real-world situations like negotiating with a vendor, outsmarting a rival company, or bluffing in a game, the AI might misjudge how far ahead to think. That's a good reason to stay alert and double-check its suggestions.