MonitoringBench: 2,644 attack trajectories expose AI monitor blind spots

A team led by monika_j has released MonitoringBench, a comprehensive benchmark for evaluating monitors that watch coding agents. The dataset contains 2,644 successful attack trajectories across four side tasks, graded by difficulty. Alongside it, the authors published a semi-automated red-teaming pipeline that decomposes attack construction into three stages: strategy generation (using human-in-the-loop, reconnaissance agents, and monitor hill-climbing), execution (with Opus 4.6 as the attacker), and post-hoc refinement (editing trajectories to reduce suspicion).

The headline finding: post-hoc refinement of attack trajectories consistently produced the lowest catch rates across every monitor evaluated—13 full-trajectory and 7 stepwise monitors. On Opus 4.5, the strongest monitor, catch rate dropped from 94.9% on directly-elicited Opus attacks to 60.3% on refined attacks. The attack taxonomy ensures broader coverage and mitigates mode collapse. Failure-mode analysis revealed four main reasons monitors miss attacks: detection failure, partial detection, detection but dismissal, and score calibration failure. The benchmark has already been used in Anthropic's Mythos risk report and OpenAI's Auto-Review evaluations.