Study Reveals Legal AI Benchmarks Overstate Performance by 85% Due to Temporal Leakage

Legal AI retrieval systems that use incoming citation context—text from later cases describing a cited precedent—have been evaluated without a temporal fence. This means at query time, the target precedent hasn't yet been cited, so any model that looks at future citations gains access to evidence it wouldn't have in deployment. The study introduces a temporal-admission decomposition that splits the naive 'relax-the-fence' gain into three components: genuine future-citation leakage, legitimate pre-query admission, and an index effect. On ECtHR-PCR (European Court of Human Rights), only 14.9% of the naive gain is actual future evidence; the rest is phantom. On CLERC (US federal, 1.84M documents), the over-credit amounts to +4.95 R@1000.

The researchers also propose a zero-training anchor channel that only admits citers dated before the query. Under the fence, this simple baseline achieves +16.1 R@1000 over BM25 on CLERC, and matches the strongest trained system on ECtHR-PCR (79.56 vs 79.39 R@1000) with zero training cost. This suggests that many sophisticated retrieval models are largely benefiting from temporal leakage rather than genuine understanding of legal relevance. The findings sound a cautionary note for AI applications in legal research, where inflated benchmark scores could mislead practitioners and regulators about real-world readiness.