BioDivergence benchmark exposes hidden contradictions in biomedical AI

Biomedical research is rife with findings that appear contradictory but are actually context-dependent—differences in cohort, geography, assay protocols, or disease subtypes can make two claims both locally valid. Existing NLI benchmarks reduce these cases to simple entailment or contradiction, missing the nuanced contextual structure. To solve this, researchers from multiple institutions introduce BioDivergence, a rigorous evaluation framework that defines a six-class conflict taxonomy and a 13-axis divergence ontology. Each claim pair is annotated with four structured outputs: conflict type, divergence axes, dominant confounder, and a reconciliation explanation.

The team releases BioDivergence-Silver-v1.0, an article-disjoint silver benchmark of 11,865 claim pairs across five biomedical domains, plus a legacy deduplicated variant for comparison. Results show notable ranking differences: a fine-tuned reference model dropped about 12 points under the article-disjoint setting, while Mistral-7B-Instruct-v0.3 achieved 0.5523 accuracy and 0.3894 contextual-F1 on the 842-example primary test set. The framework offers a more faithful way to distinguish contextual divergence from direct contradiction and to separate article-level memorization from genuine task learning, paving the way for more reliable biomedical AI systems.