Vocabulary Transfer lifts ModernBERT to 52.4 nDCG on BEIR, beating BERT-base

A new study from researchers Zhichao Geng and Yang Yang, accepted at SIGIR 2026, reveals why cutting-edge foundation models like ModernBERT lag behind the older BERT-base in learned sparse retrieval (LSR). The culprit isn't an architectural flaw but a mismatch they call the "Vocabulary Gap." Modern tokenizers prioritize lossless reconstruction, using raw, case-sensitive vocabularies that map single semantic units to multiple redundant surface forms. This wastes model capacity on morphological noise and hinders the lexical matching essential for sparse retrieval. The authors formalize this intuition with a theoretical framework showing that coarse-graining vocabularies — merging redundant surface forms — can tighten generalization bounds and improve performance, provided semantic integrity is preserved.

To bridge this gap, the team proposes Vocabulary Transfer (VT), a model-agnostic framework that adapts advanced encoders to sparse-friendly, normalized vocabularies at minimal computational cost. VT employs two key mechanisms: Semantic Initialization uses spatial topology to preserve the geometric structure of the pre-trained manifold, while Activation Potential Calibration (APC) aligns activations with sparsity constraints, preventing dead neurons and dense collapse often seen in standard fine-tuning. Empirically, VT delivers universal gains: ModernBERT achieves a state-of-the-art 52.4 nDCG on the BEIR benchmark, a +4.7 improvement over previous results. The method also resuscitates underperforming models like RoBERTa-large and extends seamlessly to inference-free architectures and specialized domains. These results confirm that the performance lag is a solvable vocabulary mismatch, not an architectural deficiency. The code and models are publicly available.