ModernBERT legal fine-tuning beats vanilla on court opinion tasks
Further pre-training on all US court opinions yields significant gains despite 500x more data.
A new paper from Dominik Stammbach and Peter Henderson explores domain adaptation of ModernBERT for legal tasks. They further pre-trained ModernBERT—which already saw 500x more data than original BERT—on all US court opinions using masked language modeling. Surprisingly, even with that massive pretraining, additional legal domain adaptation still produced significant gains on every benchmark tied to US court opinions, matching the improvements seen in early BERT domain adaptation work. Notably, training from scratch on legal data alone could not match the performance of further pre-training an existing ModernBERT checkpoint.
The fine-tuned models support sequence lengths up to 8,192 tokens, enabling them to process long legal documents. They can compute dense embeddings for legal passages, powering semantic search, or rerank hundreds of candidate passages for a given query—useful for legal research tools. The authors release all model checkpoints publicly, allowing practitioners to immediately leverage these capabilities. The work will appear at ICAIL 2026 in Singapore.
- Further pre-trained ModernBERT on all US court opinions using masked language modeling.
- Domain adaptation beat vanilla ModernBERT on every US court opinion dataset, despite 500x more base pretraining data.
- Handles up to 8,192 tokens; computes embeddings and reranks hundreds of legal passages for search queries.
Why It Matters
Public legal AI model improves document search and analysis with long-context embeddings and reranking.