Goodfire's logit monitor detects evaluation awareness 10–100x more efficiently
Reading logits at the start of a chain of thought predicts eval awareness without sampling.
Goodfire researcher Santiago Aranguri introduces a logit-based monitor for evaluation awareness that reads the model's output logits at sentence boundaries in the chain of thought to estimate the probability of producing a pre-specified eval-aware sentence (e.g., "The user might be testing"). Unlike activation probes, this method requires no training—it simply token-forces the eval sentence and multiplies token probabilities. On Kimi K2.5 and Qwen 3 32B, the logit monitor outperforms traditional verbalization monitoring (LLM judge) by a factor of 10–100x in rollout efficiency.
The logit monitor is tested on two tasks: separating evaluation prompts (Fortress and Petri) from deployment prompts (WildChat), and predicting which evaluation prompts will produce at least one rollout displaying verbalized eval awareness. Notably, the monitor already works at the very start of the chain of thought, before any rollout is sampled, and its predictive value is largely unaffected by whether the rollout actually verbalizes awareness. This suggests that the probability of verbalized evaluation awareness is primarily set by the prompt, not the chain of thought. The approach provides a cheap, interpretable signal complementary to both verbalization-based and activation-based methods.
- Logit monitor estimates probability of eval-aware sentences using token probabilities without training, outperforming LLM judge monitors with 10–100x fewer rollouts.
- Works on Kimi K2.5 and Qwen 3 32B for separating evaluation prompts (Fortress, Petri) from deployment prompts (WildChat) and predicting which eval prompts produce verbalized awareness.
- Monitor is effective even at the start of chain of thought, before sampling any rollout, and its value is largely independent of whether awareness is verbalized.
Why It Matters
Cheap, interpretable monitoring of evaluation awareness enables more reliable LLM testing without costly rollouts.