LLM medical decisions flip based on inference context, ICML study finds

Large language models are increasingly used in high-stakes decision-making, from hiring to clinical triage. Most bias research focuses on how input framing or patient demographics influence outputs. But a new paper from Spencer Gibson and colleagues at ICML 2026's AI4GOOD workshop reveals a subtler problem: the same facts can trigger different probability updates depending on whether the model sees its own previous response in the conversation history. The study asked models to assign resource-allocation probabilities to two patients given brief clinical context, then presented the same scenario again with a single extra sentence containing contrasting patient information—either with or without the model's own earlier response in the prompt.

The results were striking. In three of four tested models, the paired-context setup and the independent-inference setup produced different probability shifts, often moving in opposite directions—one favoring Person A, the other favoring Person B—despite receiving identical new information. This suggests that models aren't simply updating on new evidence; they're also anchoring to, or contradicting, their prior outputs. The authors ran additional paired-context experiments varying attributes across scenario axes to confirm the effect wasn't a fluke. For sensitive domains like medical allocation, where fairness and consistency are critical, these findings expose a dangerous fragility: an LLM's decision can depend on whether the user includes the chat history, not just on the facts presented. The paper calls for careful context engineering and more behavioral studies before deploying LLM systems in real-world decision pipelines.