AI toxicity detectors fail 35% of disabled users, study finds

A new paper on arXiv (arXiv:2607.24898) exposes critical flaws in how AI safety filters handle marginalized groups. The researchers—including experts from Microsoft—tested state-of-the-art toxicity detectors on text-to-image generated content for two disability communities: people with dwarfism and those who are blind or have low vision. They found that roughly 35% of images flagged as safe by universal detectors were actually considered harmful by members of those communities. In zero-shot settings, both large vision-language models and general-purpose detectors scored F1 below 0.40—worse than random guessing—meaning they systematically miss stereotyping, erasure, and offensive portrayals that are obvious to community members.

The team proposes community-specific toxicity detection (CTD) and demonstrates early progress using prompt-based adaptation methods. GPT-4o with in-context learning achieved F1 scores of 0.50 and 0.78 for the two communities, while parameter-efficient fine-tuning of smaller models (0.5B to 7B parameters) reached 0.48 and 0.59 using fewer than 100 examples. However, performance remains far below the ~0.90 F1 that general-purpose toxicity detectors routinely achieve. The paper notes that CTD is sensitive to evolving community guidelines and requires sustained research. The findings have immediate implications for platforms using AI image generation, suggesting that current safety guardrails may inadvertently harm the very users they're meant to protect.