AI's Secret Test-Taking Trick Could Be Fooling You

Frontier AI models can often tell when they're being evaluated—a trait called "evaluation awareness"—and that can undermine the very benchmarks used to assess them. A new open benchmark, EvalDetectBench, helps measure how reliably these models recognize evaluations and how detectable individual benchmarks are. The research also uncovers hidden biases: the model generating the comparison transcripts accounts for 11.25% of measurement variance and can even reorder model rankings, while prompts chosen for one model can perform near chance on another. EvalDetectBench corrects both issues with per-model calibration and a stratified harmonization process.