CRAwLeR: New benchmark for cross-reference aware legal retrieval

Legal document retrieval becomes far more complex when context matters — especially with cross-references that link one clause to another. To address this, researchers Maciej Jalocha and William Michelsen from the University of Copenhagen have introduced CRAwLeR (Cross-Reference Aware Legal Retrieval), a rigorous new benchmark for evaluating how well retrieval systems handle such context-dependent queries. The pipeline automatically detects legal cross-references, identifies query candidates, links target chunks to their relevant context, and uses an LLM to generate queries that truly require that context.

To ensure quality, the generated queries are filtered through both an adversarial non-contextual baseline and an assurance prompt. The result is two datasets: CRAwLeR-DK (Danish) and CRAwLeR-PL (Polish). Manual analysis shows that approximately 80% of randomly sampled queries genuinely target the labelled chunk and require context, with failures following systematic patterns. The benchmarks are hard but not solved: best Recall@10 reaches only 55% on CRAwLeR-DK and 59% on CRAwLeR-PL. Ablation studies attribute the remaining gap to the contextualizing LLM rather than the retriever itself. Notably, even when the target chunk appears in the top ten results, labeled context chunks routinely outrank it — suggesting retrievers lack true contextual understanding.

This is the first context-aware chunk retrieval dataset that carefully considers construct validity, ensuring queries actually test what they claim. The work sets a new standard for evaluating legal AI systems, highlighting that current models struggle with the nuanced cross-references that lawyers and paralegals rely on daily.