This AI Erases Race From Police Files — But Misses Key Clues

California recently told every prosecutor in the state to practice "race-blind charging" — meaning they must read case files with race-related details removed before deciding what charges to file. Doing that by hand at scale was impossible, so researchers built bc2, an AI tool (the same family of technology behind ChatGPT) that automatically blacks out those details. It has already been used in more than 119,000 real criminal cases during 2025.

The researchers then tested whether the tool actually works. Using nearly 5,000 real police reports gathered from jurisdictions across the United States, they found the newest version of bc2 correctly followed the state's rules in 96.7% of cases — a clear improvement over earlier versions and better than rival open-source tools. For a system running at that volume, that's a strong score, though it still means roughly three in every hundred reports slip through imperfectly.

The catch is that the law itself is incomplete. California's mandate does not require hiding location information — a street name or neighborhood — even though where someone lives often acts as a stand-in for race. When the tool went further and removed those extra clues too, it wiped out 43.1% of the bias-related "signal" that was still leaking through after full legal compliance. In other words, perfectly following the rules still leaves plenty of room for bias to sneak back in.

The lesson reaches well beyond California. Auditing an AI tool against the rules isn't enough — audits like this can reveal that the rules themselves fall short, and can be used to fix both the software and the policy. As governments automate decisions about jobs, benefits, and policing, independent checking is the difference between AI that genuinely reduces bias and AI that quietly launders it behind a 96.7% score.