AI legal case benchmarks reveal charge alone explains 99% of top scores

A new paper from researchers Yao Liu, Tien-Ping Tan, and Zhilan Liu reveals a fundamental construct-validity flaw in popular Chinese Legal Case Retrieval (LCR) benchmarks. The team audited three benchmarks—LeCaRDv2, LeCaRDv1, and CAIL2022—and discovered that the primary legal charge itself accounts for nearly all of the performance attributed to advanced AI models. On LeCaRDv2, a simple rule of ranking candidates by shared primary charge (with BM25 as a tiebreaker) closes 99.2% of the gap between BM25 and the best trained reranker, with no statistically detectable difference in NDCG@10. The reason: the benchmark defines top relevance via the “key constitutive elements” of the crime, which effectively encode the charge. The charge-to-relevance macro-AUC is 0.871 on LeCaRDv2, meaning charge alone predicts relevance with high accuracy. When charge is held constant, the trained reranker's advantage collapses to a tiny within-charge residual of +0.026 NDCG@10, about a quarter of the total improvement.

The effect is not uniform across benchmarks: the same charge-based rule recovers 84.3% on LeCaRDv1 and is out-of-spec on CAIL2022, where the charge-to-relevance signal weakens (macro-AUC 0.871 → 0.759 → 0.728). A predicted-charge cascade reproduces 76.6% on LeCaRDv2 but does not transfer. The team also shows that an exploratory zero-training charge-pool channel lifts LeCaRDv2 recall (R@100 +0.025) but hurts with wrong-charge controls. They release a charge-controlled evaluation (CCE) protocol—including scripts, schema, and a whitelist—to allow future benchmarks to screen for this confound before interpreting NDCG@10 as legal-reasoning ability. The paper does not claim any system actively uses charge, but rather that the benchmark design makes charge a high-leverage construct-validity factor that inflates scores.