Can Small Models Reason About Legal Documents? A Comparative Study
A 3B-parameter model activated via Mixture-of-Experts architecture matched GPT-4o-mini's accuracy on legal reasoning benchmarks.
A new research paper by Snehit Vaddi challenges the assumption that only massive language models can handle complex legal reasoning. The study, "Can Small Models Reason About Legal Documents? A Comparative Study," rigorously evaluated nine models with under 10 billion parameters across three established legal benchmarks: ContractNLI for contract entailment, CaseHOLD for legal holding identification, and ECtHR for case outcome prediction. Using five distinct prompting strategies—including direct, chain-of-thought, few-shot, and two RAG (retrieval-augmented generation) methods—the research conducted 405 experiments to identify the most effective configurations.
The most significant finding is that model architecture and training quality matter more than raw size. A Mixture-of-Experts (MoE) model that activates only 3 billion parameters per task matched the mean accuracy of OpenAI's much larger GPT-4o-mini and actually surpassed it on the CaseHOLD benchmark. Conversely, the study's largest 9B-parameter model performed worst overall, highlighting that scaling parameters alone is insufficient. For prompting, few-shot learning emerged as the most consistently effective strategy, while chain-of-thought's utility proved highly task-dependent. The research also found that for RAG, the bottleneck is the model's ability to use retrieved context, not the retrieval method itself, as both BM25 and dense retrieval yielded near-identical results.
Practically, the entire evaluation was conducted via cloud APIs for a total cost of just $62, proving that rigorous LLM benchmarking is accessible without dedicated GPU clusters. This work provides a clear roadmap for legal tech developers: smaller, well-architected models like the 3B-parameter MoE can serve as cost-effective, private, and low-latency alternatives to expensive frontier models for specific legal reasoning tasks.
- A 3B-parameter Mixture-of-Experts model matched GPT-4o-mini's accuracy on legal benchmarks and surpassed it on CaseHOLD.
- Few-shot prompting was the most effective strategy; chain-of-thought's value was highly task-dependent.
- The entire 405-experiment study cost only $62 using cloud APIs, demonstrating accessible evaluation methods.
Why It Matters
Enables cost-effective, private deployment of specialized AI for legal document analysis, reducing reliance on expensive, general-purpose frontier models.