IyawoBench v2.0 reveals AI triage failures hidden by traditional safety metrics

A new paper from Anthonio Oladimeji Gabriel and Dimeji Olawuyi takes aim at how we measure safety in AI clinical triage, particularly for low- and middle-income countries (LMICs) where physician shortages make automated triage appealing. The authors present IyawoBench v2.0, a diagnostic benchmark built from 200 synthetic vignettes derived from 1,200 real patient encounters across 19 Nigerian primary health centres. The core argument is that existing safety metrics—like a binary "did not send an emergency home" check—can show perfect scores while hiding systematic failures that make models undeployable at scale. To address this, the paper introduces a mathematical framework with fourteen definitions and two theorems that decompose triage safety into three distinct failure modes: Conservative Escalation Bias, Systematic Downgrade Bias, and Middle-Tier Instability.

The framework also proposes two new metrics: the Escalation Bias Index and Expected Deployment Cost. Evaluated on Claude Sonnet 4.6, Llama 3.3 70B, Llama 3.1 8B, and five naive baselines, the results are sobering. All three frontier models exhibit at least one formal failure mode, and traditional sensitivity metrics conceal a 77-percentage-point under-triage gap in Llama 3.1 8B—meaning severe cases were systematically downgraded to lower urgency tiers. Critically, the optimal model varies depending on the deployment scenario (Emergency-Focused, System-Sustainability, or Balanced), proving that single-ranking benchmarks are inadequate for selecting clinical AI in LMIC settings. This work provides both a public benchmark and a transferable diagnostic framework for any triage-style clinical AI evaluation.