Self-evolving LLM agent beats human rules for legal case retrieval

Legal case retrieval remains notoriously difficult due to complex jargon and the need for precise lexical matching. While dense retrieval models have made progress, BM25 still serves as a surprising strong baseline. To exploit this, the authors introduce a self-evolving agent that enhances BM25 via rule-driven query rewriting – without any parameter training. The LLM-based agent is equipped with an automatic evaluation environment, allowing it to iteratively create candidate rewriting rules, design validation experiments over rule combinations, and prune ineffective rules based on historical feedback. This closed-loop process lets the agent improve its rule set autonomously, much like an AI researcher running experiments.

Evaluated on the Chinese legal retrieval benchmark LeCaRD-v2, the framework outperforms non-evolutionary baselines, including human-designed rules and greedy rule selection. The benefits are most pronounced when using a high-capacity core LLM. Detailed analyses reveal that the LLM's ability to leverage previous experimental results and its intrinsic knowledge of rule elimination are critical for effective self-evolution. The paper will be presented at ACL 2026 and offers a practical way to boost legal search accuracy without expensive retraining or manual rule engineering.