The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles

A new arXiv paper (2605.01407) by Hiun Kim, Tae Kwan Lee, and Taeryun Won investigates how pre-training data and hyperparameters affect Expanded-SPLADE (ESPLADE) models when fine-tuned for neural information retrieval (IR). ESPLADE, a variant of the SPLADE family, leverages mask language modeling (MLM) at both pre-training and fine-tuning stages. The team used in-house web document titles as the specific dataset, simulating a realistic scenario where search indexes rely on title-only text.

Their experiments reveal three counterintuitive findings. First, models pre-trained on a general corpus (not title-specific) with a higher learning rate achieved superior retrieval effectiveness in both unpruned and most-strict pruned settings, even though those models exhibited lower MLM accuracy. Second, when sparse vectors are aggressively pruned, the most effective models incur higher retrieval cost and greater variance in per-term posting list lengths. Third, simply repeating the general pre-training corpus does not meaningfully improve downstream performance. These results empirically demonstrate the trade-off between retrieval cost and effectiveness under strict pruning, and underscore the difficulty of aligning MLM pre-training objectives with sparse retrieval fine-tuning.