New MLAIRE protocol reveals hidden flaws in multilingual search models

A team of researchers (Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim) have introduced MLAIRE (Multilingual Language-Aware Information Retrieval Evaluation Protocol), a new evaluation framework that goes beyond traditional semantic relevance metrics for multilingual search. Current benchmarks treat results equally regardless of language, but real users often prefer content in their query language for readability and verification. MLAIRE constructs controlled pools of parallel passages across languages, enabling separate measurement of semantic retrieval accuracy and query-language preference. It proposes two novel metrics: Language Preference Rate (LPR) and Lang-nDCG, along with a 4-way failure decomposition that distinguishes semantic errors from language-preference errors.

The team evaluated 31 different retrievers including dense, sparse, and late-interaction models. Their findings show that standard metrics hide significant behavioral differences: some retrievers are semantically strong but tend to return correct content in a non-query language, while others prioritize query-language matching at the expense of semantic relevance. This has direct implications for retrieval-augmented generation (RAG) systems, where language mismatch complicates downstream grounding and answer verification. MLAIRE provides a more nuanced toolkit for developers building multilingual search products, ensuring users get results they can actually read and trust.