LLMs fail at deception in Secret Hitler, study finds

Researchers at the University of Göttingen benchmarked several large language models—including Llama 3.1 70B—on their ability to deceive, persuade, and reason strategically in the hidden role game Secret Hitler. The study introduced an open-source framework with three new metrics: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. Results showed a sharp gap between conversational fluency and strategic depth. Models playing as Fascists (the deceptive role) consistently produced negative impact scores and failed to sustain lies, leading to games roughly 40% shorter than human matches.

Surprisingly, reasoning-enhancement techniques like Chain-of-Thought prompting and internal memory did not improve performance—Fascist roles actually experienced up to a 23.2% decrease in win rates. In contrast, simple rule-based agents aligned with expert human voting decisions 86.7% of the time, while Llama 3.1 70B managed only 59.7% accuracy. The authors conclude that current architectures remain ineffective at complex, multi-turn manipulation, and suggest the framework as a reproducible testbed for detecting when future models begin to master deceptive behaviors—a critical capability for AI safety.