New AI 'Safety Judge' Catches Dangerous Chatbot Replies That Big AI Misses
Chatbots can give harmful advice—this new tool spots more crisis signs.
When you use an AI chatbot for mental health advice, who makes sure its response isn't harmful? Big AI companies often use other AI models to check. This paper tested that idea by having three top models (GPT, Claude, Gemini) judge thousands of AI messages about mental health, companionship, and coaching, comparing them to ratings from a real psychologist.
The results were troubling. The AI judges disagreed strongly, especially on empathy—where polite but empty phrases can hide real problems. Worse, Gemini was the most lenient judge: it called a response that suggested harming oneself "exemplary." Averaging the judgments, a common fix, just blended these mistakes into the final score.
The researchers then created aipsy-judge, a smaller, open-source AI model fine-tuned on the psychologist's corrections. It catches 92% of crisis situations, has higher agreement with human experts than the big frontier models, and works locally—so your private chats never leave your device. It's not perfect (it errs toward over-flagging, which is safer), but it shows that safety checking can be both more accurate and more private.
- Top AI judges like Gemini are dangerously lenient—one even rated a self-harm response as 'exemplary.'
- The new aipsy-judge model catches 92% of crisis situations and runs locally on your machine.
- Using a psychologist's ratings to train a smaller model beats simply averaging big AI opinions.
Why It Matters
This could make AI mental-health support safer and more private, protecting vulnerable users from harmful advice.