AI That Finds Court Cases Missed Most of Them in Public Test

A university research team entered an annual contest called COLIEE 2026, where computer programs try to find the past court cases that a given legal case refers to. That sounds simple, but the contest hides the exact quoted passages, so the AI can't just match words. Their system chained four tools together: a keyword search, an AI re-ranker, a "knowledge graph" (a map of how cases, people and legal ideas connect), and a final scoring model that weighed 34 different signals.

On the real test, their system scored 0.177. On their own internal tests, it had scored 0.311 — nearly twice as good. That kind of gap is the quiet nightmare of AI work: a system that looks impressive in the lab and stumbles in public. The team didn't just shrug. They ran experiments to test their own excuses one by one, checking whether the problem was timing, bad thresholds, or the AI simply never finding the right case in the first place.

The answer was split exactly down the middle. Half the misses were cases the system never found at all. The other half were cases it found but then ranked too low, below the cutoff. Fixing the first half worked: tuning the keyword search and adding richer search views lifted how often the right case appeared in the top 200 results by three to seven points. Fixing the second half mostly failed — date filters, fancier re-rankers and per-query rules did nothing.

They also documented four evaluation mistakes in their own work, each of which had flipped a conclusion until corrected. One was "leakage": the AI was sneakily being tested on information it had already seen during training, which made it look better than it was. The honest takeaway is that legal AI is still far from reliable — and that AI performance numbers, including the ones companies advertise, can be surprisingly easy to get wrong.