New Paper: AI's '90% Reliable' Promises Are Often Wrong

When an AI system decides whether to answer a question, flag a risky email, or block harmful content, it often uses a 'threshold' set during testing. For example, a system might promise to be 90% accurate by drawing the line at the 90th percentile of its calibration data. That promise sounds solid — but it quietly depends on an assumption: that every test example is independent, like flipping a coin each time.

In modern AI, that assumption is violated. Many examples in a calibration set share the same underlying prompt, document, or reasoning trace. They're not independent; they're clustered. Imagine polling 25,000 people but accidentally interviewing the same 1,300 families over and over. You'd think you had a huge sample, but you really have much less. Statisticians have known how to fix this for averages since 1965, but thresholds behave differently.

Adam Noonan, the paper's author, proved that the correction currently used in AI research is wrong — and can miss in either direction. He also showed that a dataset has no single 'true size.' It depends on where you set the threshold. The effective sample size changes at every level. In a real calibration set of 25,028 examples, the reliability was closer to that of just 1,300 genuinely independent examples. That's a massive difference.

The scariest part? The damage is invisible when you average over many test runs. It only hits you when you deploy the AI once — in a real product, a real decision, a real life. So next time an AI says it's 99% sure, ask: sure about the average, or sure about me?