New 'Same Game' Benchmark Shows LLM Agents Fragile to Framing Shifts

A new paper from researchers at an undisclosed affiliation (arXiv:2607.19670) proposes "Same Game, Different Story," a minimal benchmark for evaluating the strategic robustness of large language model (LLM) agents. The benchmark defines strategic robustness as the invariance of a model's action distribution under payoff-preserving changes in framing—meaning the same game, told with different stories, should yield the same choices. The authors illustrate their framework by reanalyzing published data from GPT-3.5, GPT-4, and LLaMa-2 across four classic social-dilemma games (e.g., Prisoner's Dilemma, Chicken), comparing business-oriented and friend-sharing framings. The dataset covered 24 model-game-context cells and 7,200 individual decisions, though trial-level data had to be reconstructed from aggregate figures.

The results reveal significant framing sensitivity. Under the paper's conservative transformation, the pooled strategic robustness score across all models and games was 0.783 (on a scale where 1.0 would indicate perfect consistency). Friend-sharing framing increased cooperation rates by an average of 0.307 relative to business framing—a 30.7% shift—demonstrating that social-relational narrative can dramatically alter LLM behavior even when underlying incentives and action sets are identical. The authors argue that strategic robustness must be evaluated separately from strategic competence (ability to play optimally), and recommend using families of payoff-equivalent prompts rather than a single game presentation. This work has direct implications for deploying LLM agents in negotiation, pricing, or any multi-agent context where consistent strategic behavior is critical.