DECODEM benchmark shows LLMs can accurately extract corporate governance data
New benchmark reveals frontier models parse complex legal documents with near-perfect accuracy.
Jens Frankenreiter from arXiv introduces DECODEM, a benchmark for automated extraction of corporate governance variables from organizational documents. Traditional legal research relies on costly, opaque human coding of charters and bylaws. DECODEM pairs randomly sampled corporate charters with high-quality human annotations covering a range of governance provisions, providing standardized datasets for evaluating LLM extraction pipelines. The study tests several LLM approaches varying in prompt design, task decomposition, and document handling, treating extraction as a set of binary classification problems.
Results show automated extraction is feasible with high accuracy for many provisions—median performance nears the upper bound across all models. However, performance varies systematically, with a few provisions causing most errors. Interestingly, more elaborate prompting and cascading pipelines don't consistently improve frontier models but substantially narrow the gap between frontier and efficiency-oriented models. This suggests pipeline design can partly substitute for model capability. DECODEM demonstrates that current frontier LLMs can extract legally meaningful information from complex corporate documents with high accuracy, paving the way for automated feature extraction in corporate governance dataset construction.
- DECODEM provides benchmark datasets pairing randomly sampled corporate charters with human annotations for multiple governance provisions.
- Median extraction accuracy across LLM pipelines approaches the upper bound, with most errors concentrated in a small number of provisions.
- Elaborate prompting and cascading pipelines close the performance gap between frontier and efficiency models, but don't consistently boost frontier models.
Why It Matters
Automates costly, opaque human coding of legal documents, enabling scalable and transparent corporate governance research.