SHIFT method eliminates language bias in multilingual search — no retraining needed
A simple vector subtraction during indexing fixes multilingual search bias — no model retraining required.
Multilingual dense retrieval models often favor documents in the same language as the query, creating severe language bias that pushes more relevant foreign-language results down the rankings. To address this, researchers from Korea University propose SHIFT (Semantic Harmonization via Index-side Feature Transformation), a training-free technique applied entirely during the document indexing stage.
SHIFT leverages parallel translation pairs (aligned sentences across languages) to estimate a relative language vector for each target language with respect to the source language. During indexing, it subtracts this vector from document embeddings to neutralize language-specific offsets. The method requires no model retraining, extra inference steps, or changes to the retrieval pipeline. Evaluated on 4 MLIR benchmarks (including XOR-Retrieve and MIRACL) and multiple dense retriever backbones (e.g., Contriever, LaBSE), SHIFT consistently reduces language bias and improves cross-lingual retrieval performance — a practical, lightweight fix for global search systems.
- Training-free method applied during document indexing, not inference or training
- Uses parallel translation pairs to compute relative language vectors for each target language
- Tested on 4 MLIR benchmarks with multiple dense retrieval models; effectively reduces language bias
Why It Matters
Enables fair, cross-lingual search without expensive retraining — a practical boost for global information access systems.