AI Safety Filters Miss the Most Dangerous Requests, Study Finds

AI chatbots come with safety monitors — think of them as bouncers checking every request at the door. These bouncers are supposed to spot harmful questions and turn them away before the chatbot answers. But how do we know if they're doing a good job? Usually, we test them on a list of known-bad prompts and check how many they catch. That number, called "recall," is how AI companies measure safety.

The problem, as this new study shows, is that catching a bad prompt only matters if the chatbot would have actually answered it. The researcher ran thousands of harmful prompts through several different AI models many times. He found that some prompts sometimes get a harmful answer and sometimes don't — it's not predictable. He called these prompts "elicitable." Then he looked at whether the safety monitors were better at catching elicitable prompts or non-elicitable ones.

The results are worrying. Across three different models and six safety monitor designs, the monitors' recall was 22% to 38% lower on the prompts the model actually complied with. In other words, the very requests most likely to produce a harmful answer are the ones the bouncers miss. Prompts the monitors missed were 2.8 to 5.6 times more likely to be answered harmfully than the prompts they caught. This held true even for text-only monitors that didn't use the target model at all.

So the safety ratings we hear about are inflated. Companies may think their filters are stopping dangerous content, but they're systematically failing on the most dangerous cases. The paper suggests a better approach: don't just test filters against a list of bad words or labels. Test them against what the actual model does when faced with a real request. Until that happens, we can't trust safety numbers — and that matters for anyone using AI tools for everything from homework help to customer service.