New Study Shows Evaluation Design Skews Expert vs. Auto MeSH Comparison

A new preprint from Samuel Okoe-Mensah (arXiv:2607.21685) tackles a practical problem in systematic reviews: classifiers that prioritize relevant abstracts often use Medical Subject Headings (MeSH) as input features. These MeSH terms come either from expert indexers (weeks or months after publication) or from automatic tools (immediately). Surprisingly, no prior work had directly compared expert vs. auto MeSH as classifier features, nor asked whether the comparison outcome depends on evaluation design. Using the Cohen et al. (2006) drug-class benchmark on three topics (Statins, Opioids, ADHD), the author compared a bag-of-words logistic regression classifier (seven reruns) with BiomedBERT (five seeds), both using either expert or automatic MeSH.

The results reveal that evaluation design dramatically changes conclusions. Under the canonical 5-fold full-corpus cross-validation on Statins, the bag-of-words expert-vs-auto gap was +0.096 WSS@95%, strongly favoring expert MeSH. But when the corpus size was matched to the smaller topics (n=803), the gap shrank to +0.033 (95% bootstrap CI includes zero). Using 10-fold instead of 5-fold at full size gave +0.021 (CI barely excludes zero). BiomedBERT under canonical evaluation showed a gap of just +0.020, within sampling noise of the bag-of-words 10-fold result. A power analysis revealed that Opioids and ADHD had too much variance to detect a Statins-sized effect, meaning those null results are design-limited. Additionally, 15.1% of Statins inputs exceeded BiomedBERT's 512-token limit when expert MeSH were appended, suggesting truncation may partially explain the smaller transformer gap. The paper concludes that benchmark conclusions about feature sources can shift substantially under reasonable evaluation design changes, urging researchers to standardize or transparently report evaluation designs.