AI Now Checks If Laws It Reads Are Trustworthy
Your next legal AI could make costly mistakes—here's how researchers hope to stop that
A computer scientist just published a way to ask: *Can we trust what AI thinks a law says?*
Right now, AI is often used to scan laws, contracts, or regulations and pull out key rules. But when two different AI tools read the same law—like Missouri’s rules on minimum age for marriage—they don’t always agree. One might miss a key number entirely, or misread a threshold. That uncertainty matters a lot if the AI is being used to draft legal advice or check compliance.
So, Surya Saka built a kind of “trust test” for AI legal reading. It runs the AI’s output through thousands of simulated checks to see which conclusions hold up under noise or disagreement. Only the most reliable rules get a green light. When tested on 29,000 sections of Missouri law and 500 Indian acts, the test passed on most laws—but only if the AI was carefully set up for each chapter. Otherwise, over 90% of chapters failed the trustworthiness test.
The bottom line: AI can help read laws, but without careful calibration, it can still be wrong most of the time. The researcher released all the tools and data, including one mistake that was later fixed—showing how transparency can help catch errors before they cause real problems.
The system isn’t plug-and-play—it needs tuning, and even then, it’s fragile. But it’s a step toward making AI safer when it touches legal text.
- AI often misreads laws or misses key numbers when scanning legal text
- New system tests whether AI’s conclusions about laws are reliable under real-world noise
- Works only if AI is carefully calibrated per chapter—otherwise, most laws still look unreliable
Why It Matters
AI could soon help draft or check legal documents—this work is the first step to making sure it doesn't make costly mistakes