AI Legal Benchmarks Miss the Real Problem: Messy Pro Se Inputs

A new paper by Andrew Lou and David Shin, accepted at the AI4Law Workshop at ICML 2026, challenges a core assumption of legal AI research: that current benchmarks can predict how well LLMs will improve access to justice for people representing themselves (pro se litigants). The authors argue that existing benchmarks like LEXam evaluate models on inputs that have been preprocessed and curated by legal experts—what they call the 'upper bound' of performance. In reality, pro se litigants submit prompts filled with noisy narratives, buried facts, omissions, folk-legal assumptions, and surface-level errors. These degradations mirror conditions known to cause LLM failures, such as long-context sensitivity, underspecification, hallucination, and typographical perturbations.

To demonstrate the gap, the authors conducted a small perturbation experiment on LEXam, introducing pro se-like errors into benchmark queries. The results showed a measurable drop in model accuracy, confirming that the 'lower bound' of performance is far below the upper bound. They connect this evidence with established machine learning research on model degradation under noisy inputs. The paper concludes by urgently calling for new legal benchmarks that directly measure robustness under pro se-like inputs, so that claims about legal AI improving access to justice can become empirically testable. Without such benchmarks, the gap between lab performance and real-world utility may remain hidden or even widen.