Legal AI reliance lacks evidence, new paper warns

A new paper by James Bryan Williams reveals a critical mismatch between AI capabilities and legal system requirements. Analyzing over 1,500 New York court cases involving AI hallucinations in legal documents, Williams found courts routinely demand error rates and performance benchmarks that simply don't exist. Instead, the legal system has defaulted to process-based solutions like training mandates and uncalibrated human review.

The paper identifies four key contributions: mapping legal duties to human-computer interaction concepts, extracting official requirements from court records, analyzing the substitution of process for evidence, and proposing a research agenda for computing. Williams argues that without concrete error metrics and standardized benchmarks, the justice gap risks widening rather than closing as AI tools become more prevalent.

The study highlights how legal aid programs bear the heaviest burden, forced to track their own error rates while judges improvise their own testing methods. Williams calls for task taxonomies, shared error metrics, maintained benchmarks, and private data evaluation harnesses to properly ground appropriate reliance on legal AI.