Nepali Legal AI: RAG-based QA achieves 91% precision with BM25

A team of Nepali researchers (Wagle, Adhikari, Khanal, et al.) have introduced the first Retrieval Augmented Generation (RAG) framework for legal question answering in the Nepali language. The system leverages case law documents extracted from the Nepal Kanun Patrika digital archive. For document retrieval, they compared BM25, a classical keyword-based method, with the multilingual E5 large embedding model. BM25 achieved a top precision (P@1) of 91%, significantly outperforming E5's 75%. The retrieved documents were then fed into a generative model to produce answers to legal queries.

The evaluation focused on both retrieval and generation quality. Generated answers achieved 74% groundedness (how well they were supported by retrieved documents) and 85% truthfulness as judged by an automated metric. Human evaluators rated truthfulness at 84%, and the overall answer generation success rate was 92%. These results demonstrate that a RAG pipeline can effectively address data scarcity in low-resource languages. The framework provides a reliable foundation for deploying AI in the Nepali legal domain, potentially democratizing access to legal information for millions of Nepali speakers who previously lacked such tools.