Simple Rescaling Trick Boosts SPLADE Retrieval with ModernBERT

Learned sparse retrieval (LSR) models like SPLADE have traditionally relied on BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT with stronger pretrained encoders should improve retrieval effectiveness. However, researchers found that under standard SPLADE training recipes, backbones with large MLM-head L2 norms (e.g., ModernBERT, Ettin) can suffer performance degradation and even training collapse. They identified the root cause as a scale mismatch: SPLADE directly uses MLM-head outputs to build sparse lexical representations, and query-document relevance is computed via an unnormalized dot product. An inflated MLM-head scale amplifies sparse activations, distorts matching scores, and destabilizes contrastive training.

To address this, the team introduced a simple initialization-time correction that rescales the MLM-head projection by a constant factor before SPLADE training. This zero-cost adjustment improves training stability without modifying the model architecture or training objective. Across both in-domain and out-of-domain retrieval benchmarks, the correction turned unstable training runs into competitive sparse retrievers. In several settings, the corrected ModernBERT and Ettin models matched or surpassed the classic BERT-SPLADE baseline. These findings suggest the real bottleneck in adapting pretrained encoders to LSR is not encoder capacity, but the calibration of the MLM-head scale used to construct sparse lexical representations.