LLM agent audit reveals belief tracking boosts Werewolf wins by 90%
Active-belief condition raised good-side win rate from 20.5% to 39% in 1,080 games.
In a new preprint, Gao, Yang, Zhao, and Zhang introduce a rigorous audit framework for evaluating LLM agents in hidden-information social deduction games, specifically the 9-player variant of Werewolf. Agents operate under strict code-level information isolation about hidden roles. The framework maintains an external belief state, logs belief updates and action-belief deviations, and supports a defensive offline improvement loop. This allows researchers to replay and review bad cases before changing strategies.
Across 1,080 frozen games across multiple conditions, the active-belief condition was associated with a substantial improvement in good-side outcomes. In the key 200-seed paired A0/A1 comparison, the good-side win rate jumped from 0.205 to 0.390 (McNemar χ² = 16.4, p < 0.001). Surprisingly, direct action-belief consistency was low (≈0.21), and giving belief only to the werewolves helped the good side more than giving it only to the good side. The authors emphasize that the main contribution is the audit framework itself, which makes effects measurable and turns opaque agent behavior into replayable evidence for safer, controlled iteration.
- Active-belief condition improved good-side win rate from 20.5% to 39% in 200-seed comparison (p<0.001)
- Direct action-belief consistency was low (~0.21), suggesting mechanism is not simple belief-following
- Audit framework logs belief updates and deviations, enabling replay-based defensive improvement loops
Why It Matters
New auditing framework enables safer, controlled iteration of LLM agents in high-noise multi-agent settings.