TW-LegalBench: New benchmark exposes LLMs' legal citation flaws

Researchers from Taiwan have released TW-LegalBench, a comprehensive benchmark designed to measure large language models' understanding of Taiwanese law. The benchmark fills a gap in civil-law evaluation, which has historically focused on English common law or simplified Chinese sources. It comprises three tasks: over 16,000 multiple-choice questions sourced from five years of official examinations across 18 legal domains, 117 open-ended essay questions with official scoring rubrics, and more than 14,000 legal judgment prediction instances covering hundreds of crime categories. The dataset leverages Taiwan's publicly available legal corpus to provide a rigorous test of both factual recall and reasoning.

Thirteen LLMs were evaluated using accuracy for multiple-choice questions, a decomposed LLM-as-Judge framework for essays, and metrics for sentencing accuracy and statute citation in judgment prediction. Results showed that top-performing models could exceed the passing threshold for qualified lawyers (11% pass rate) but fell short of that for judges and prosecutors (1-2% pass rate). While models demonstrated reasonable verdict type accuracy and sentence prediction capability, they struggled to cite exact legal articles. These findings underscore that reliable legal text generation remains a significant challenge for LLMs, even as their performance on qualification exams approaches human levels. The work will appear at ICAIL 2026.