BioBERT outperforms LLMs in InferBERT pharmacovigilance – domain expertise beats scale

A new study from researchers Csaba Kiss, Roland Molontay, and Gabriele Pergola (arXiv:2606.17113) rigorously evaluates how model choice impacts causal adverse drug event (ADE) detection within the InferBERT framework — a system that combines transformer models with Do-calculus to distinguish true side effects from spurious correlations. The team compared four models: XGBoost (baseline), ALBERT (original InferBERT), BioBERT (biomedical transformer), and Med-LLaMA (a medical LLM). Using 5-fold cross-validation over 20 runs on two pharmacovigilance benchmarks — Analgesics-induced Acute Liver Failure (AILF) and Tramadol-related Mortalities (TRAM) — they measured accuracy, calibration error (ECE), and Jaccard concordance with traditional disproportionality signals (PRR, ROR, EBGM).

Results show that BioBERT significantly outperformed all other models on both datasets, achieving the highest accuracy and strongest concordance with established pharmacovigilance signals. Surprisingly, Med-LLaMA — despite its massive size and parameter-efficient fine-tuning — yielded the worst performance among the transformer models. Post-hoc calibration (isotonic regression) improved calibration error but had inconsistent effects on accuracy and causal discovery. The key takeaway: domain-specific pre-training provides a clear advantage over both simpler baselines and larger LLMs. For computational pharmacovigilance, investing in manageable, domain-aware models is far more effective than simply scaling model size. This challenges the current trend of defaulting to the largest available LLM for specialized scientific tasks.