Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions

A new paper from Sobotka, Karabag, and Topcu (UT Austin) dissects why large language models (LLMs) repeatedly fail at strategic tasks like negotiation and policymaking. Testing open‑weight models Llama 3.1, Qwen3, and gpt‑oss in classic incomplete‑information games, the authors uncover two fundamental disconnects. First, an observation‑belief gap: LLMs encode internal beliefs about hidden game states that are substantially more accurate than their own verbal reports. However, these beliefs are fragile—accuracy degrades with multi‑hop reasoning, shows primacy/recency biases, and drifts from Bayesian coherence over long interactions. Second, a belief‑action gap: the implicit conversion of those internal beliefs into actions is weaker than using externally prompted beliefs, but neither approach consistently yields higher payoffs. The findings suggest that even when LLMs “know” the right thing, they often cannot act on it.

These results have direct implications for deploying LLMs in high‑stakes strategic domains. The brittleness of internal beliefs—susceptible to reasoning chains, order effects, and drift—means that longer or more complex games will exacerbate mistakes. The belief‑action gap further implies that simply improving an LLM’s factual accuracy won’t fix its strategic decision‑making; the link between knowledge and action must be strengthened. The authors argue for robust guardrails and deeper analysis of internal processes before relying on LLMs for autonomous negotiation, resource allocation, or adversarial reasoning. This work also opens new research directions: designing architectures that maintain Bayesian coherence and building action modules that better leverage the models’ own hidden beliefs.