MafiaScope probes LLM beliefs in social deduction games

Researchers at the intersection of NLP and multi-agent systems have long struggled to peek inside the 'black box' of LLM reasoning during social interactions. MafiaScope, a new open-source testbed by Ilia Karpov, turns the classic social deduction game Mafia into a precise measurement instrument. The key innovation: after every public utterance, each agent privately answers a configurable set of structured probe questions (e.g., "Who do you think the Mafia is?"). These answers never re-enter the game and are automatically scored against ground truth known only to the engine. This non-invasive technique captures time-resolved belief trajectories without contaminating the agent's behavior.

In a case study using DeepSeek across 32 games (13,815 parsed probe answers), MafiaScope revealed significant calibration issues: stated confidence had an expected calibration error of 0.17, and agents over-predicted being suspected by other players 1.5 times more often than reality. The tool also includes an interactive visualizer with an impersonate mode (showing the game from one agent's perspective), timeline-aligned accuracy charts, and a counterfactual replay system that can fork any recorded step—demonstrated in a 30-fork experiment. The entire engine, viewer, and a corpus of 200+ cross-model games are released under an open license.