Clinical LLMs show 'narrative anchoring': same facts, different diagnoses

A new arXiv paper from researchers Prabhjot Singh, Pritam Deka, and Vijay Chennareddy exposes a troubling blind spot in clinical AI: large language models don't just reason over medical content — they're swayed by the way patients tell their story. Dubbed "Narrative Anchoring," this failure mode causes diagnostic outputs to diverge when identical clinical facts are expressed in different sociolinguistic registers, with no demographic markers involved.

The team built a benchmark of 1,000 USMLE clinical vignettes, each rewritten into three distinct personas (e.g., different education levels, speech patterns, or narrative styles) while preserving the underlying medical facts. Fact preservation was independently audited by a separate model that never saw the generation prompts. Across seven language models spanning three architecture families, every model showed statistically significant anchoring, with a Narrative Anchoring Gap between 0.064 and 0.151.

Standard mitigation methods proved insufficient: chain-of-thought reasoning and explicit debiasing instructions only partially reduced bias, and their apparent gains were often confounded by accuracy collapse. To address this, the authors introduce NarrativeShield, a three-agent pipeline that structurally extracts and verifies clinical facts before diagnostic reasoning begins. This approach reduces the anchoring gap to near-zero (-0.004 to 0.037) and achieves the lowest rate of severely unstable decisions of any method tested, at a modest and mechanistically expected accuracy cost for most models.

The paper also includes a stress test showing that a non-instruction-tuned base model fails to execute debiasing interventions altogether, revealing that zero-shot instruction-following ability — not just prompt content — gates the effectiveness of such methods. The dataset, human-validated for fact preservation, is released as a standalone resource for studying register-based clinical bias.