Researchers Want Legal AI to Stop Inventing Law That Doesn't Exist

Legal AI tools are increasingly used to draft contracts, answer tenant questions, and research cases. The problem: they sometimes "hallucinate" — a polite way of saying the AI confidently makes things up, including court cases that never existed. Two researchers, Maksym Taranukhin and Vered Shwartz, published a position paper saying everyone is grading these tools on the wrong test. Right now, most checks simply ask whether a cited case or law is real.

The authors say that's not enough. Imagine an AI telling you a rule applies to your situation. It cites a real, famous case — so it passes the "is this real?" test. But that case might have been overturned, or it might be from another state or country, or the law might have changed last year, or the case might not actually support what the AI claimed. Real citation, worthless advice. You could lose money, sign a bad lease, or miss a deadline because of it.

Their fix is a concept they call "legal warrant" — a fancy term for: is this claim properly backed by law? To count, the authority has to exist, apply to your jurisdiction, be current as of today, have the legal status the AI claims, and actually support the point being made. They also argue good legal AI should sometimes refuse to answer, ask you clarifying questions, or warn you about a false assumption — and those behaviors should be rewarded, not penalized.

They lay out a specific scoring system: how to record claims, label support, weight risky answers, and report how reliably human reviewers agree. They ran a small pilot test to show it works. The catch: this is a research agenda, not a product, a law, or a regulation. Nobody is required to adopt it. But it signals growing pressure to make legal AI honest — and until that happens, treat any AI legal answer as a rough first draft you verify with a real lawyer.