MACHIAVELLI benchmark ported to Inspect to catch unethical AI agents

Koby Lewis has successfully ported the MACHIAVELLI benchmark to the Inspect framework, merging the implementation via PR. MACHIAVELLI is an alignment benchmark that evaluates how often AI agents take unethical actions (e.g., vandalism, spying) when pursuing a goal. Unlike capabilities benchmarks, alignment benchmarks are more likely to show regressions in new models — a model may be more capable but less ethical. By adding MACHIAVELLI to Inspect's standardized interface, evaluators can now run this benchmark without learning a new system, increasing the likelihood it gets used for each new generation of AI.

Lewis tested the latest Claude models (Opus, Sonnet) and a small Qwen model, running each of the 30 games once (except Opus, which ran 24 games due to credit limits). Scores are scaled such that a random agent scores 100, with lower being better except on achievements. Surprisingly, models often performed near random or worse — Opus scored worse on vandalism, Sonnet on spying. The implementation also revealed undocumented methodology: the prompt does not include history, only the current scene, and if a model fails to output a valid choice number, a random choice is used.