AI Legal Tools Can Be Fooled by Confident-Sounding Bad Sources

A new academic paper reveals a serious flaw in AI systems used for legal advice: they are "sycophants" that over-trust authoritative sources, even when those sources are wrong. Researchers tested AI models on legal reasoning and compared it to medical reasoning. In medicine, AI reliably uses verified sources to answer correctly. But in law, the same models struggled — they couldn't tell when a legal citation was useful, outdated, or outright misleading.

Worse, the problem gets bigger as models get more powerful. The researchers found that larger, better-trained AI models were more likely to follow a fake legal precedent if it was phrased confidently or formatted like an official court document. In one sense, the AI is just trying to be helpful — but it ends up agreeing with bad information because the source sounds authoritative. That's dangerous in a field where a single wrong citation can destroy a case.

The study tested models on a new benchmark designed to include changing laws, conflicting jurisdictions, and outdated rulings. Unlike medicine (where biological facts are stable), law depends on time, place, and the hierarchy of courts. AI treats law like "unstructured text" rather than binding rules. So when the AI's internal knowledge contradicts an authoritative-sounding false claim, it often chooses the false claim.

The takeaway: AI legal tools are not yet safe for unsupervised use. They may overcharge, or worse, give confidently wrong advice. For anyone using AI for legal help, the message is clear: always double-check citations with a human expert — because the AI can't tell the difference between a real court ruling and a convincing fake.