Prompting beats fine-tuning: New method tops legal term retrieval benchmarks

In a new paper accepted to the ASAIL Workshop at ICAIL 2026, researchers Alvin Wang and Jaromir Savelka tackle the problem of statutory term retrieval—ranking case-law sentences by how useful they are for explaining vague legal terms in U.S. statutes. They use an established dataset of 26,959 sentences from 42 U.S. Code concepts, each labeled into four explanatory-value categories. The team compares two families of methods: supervised fine-tuning of encoder-only models (ModernBERT) and zero-shot prompting of decoder-only models.

Surprisingly, zero-shot prompting of decoder-only models achieves the strongest overall effectiveness across all NDCG cutoffs. Their best system, using a generative expected value scoring approach, surpasses all previously reported state-of-the-art results on this task. The findings directly challenge the assumption that domain-specific fine-tuning is necessary for high-quality legal information retrieval, suggesting that modern LLMs can handle nuanced legal reasoning without additional training data or computational overhead.