STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval

A team of Algerian researchers has introduced STCALIR, a novel framework designed to tackle a critical bottleneck in legal AI: the costly and time-consuming process of creating test collections for information retrieval models. In specialized domains like Algerian law, high-quality corpora and human-annotated relevance judgments are scarce, making it difficult to train and evaluate AI systems. STCALIR addresses this by generating semi-synthetic test collections directly from raw legal documents, following the established Cranfield paradigm but automating its core components—topics, corpus, and relevance judgments. The multi-stage retrieval and filtering pipeline slashes the manual annotation workload by an impressive 99%, transforming a process that could take months into one that takes hours.

The validation results are compelling. When tested against the Mr. TyDi benchmark, the semi-synthetic relevance judgments produced by STCALIR yielded retrieval effectiveness nearly on par with human-annotated evaluations, achieving a Hit@10 score of approximately 0.785. More importantly, the rankings of different retrieval systems based on STCALIR's labels showed strong statistical agreement with rankings based on human judgments, with a Kendall's τ of 0.89 and a Spearman's ρ of 0.92. This demonstrates that the framework doesn't just save time; it produces reliable, reproducible evaluations. The work, detailed in an arXiv preprint, provides a blueprint for building cost-efficient, high-quality test beds in other low-resource legal and technical domains globally, accelerating the development of region-specific legal AI tools.