LLMs triage low-SES patients to ER more – Gemini infers from ZIP code alone
Three models show 13-50pp higher ER referrals for poor patients from identical symptoms.
Researchers held a single neurological symptom profile constant and varied only socioeconomic status (SES) signals—explicit (insurance, occupation, housing) and implicit (a US ZIP code). Three deployment-tier models (Gemini 3.5 Flash, Claude Sonnet 4.6, GPT-5.4-mini) all raised emergency-room referral rates for lower-SES patients given explicit signals (spreads of 13–50 percentage points). The effect is protective (more care for poorer patients) but clinically invisible: models' reasoning traces remained nearly identical across conditions.
Critically, only Gemini 3.5 Flash inferred SES from a five-digit ZIP code alone, shifting its ER rate by 11.4 points (p=1.4e-7, same direction in 6/6 pairs). Claude Sonnet 4.6 stayed flat (-0.1 points) and GPT-5.4-mini showed a small, inconsistent shift (2.0 points, only 2/6 pairs match predicted direction). A single-sentence system prompt reduced Gemini's ZIP-code gap to 5.8 points, but did not eliminate it. The authors release all code, prompts, and raw results.
- All three models (Gemini Flash, Claude Sonnet, GPT-mini) raised ER referrals 13–50 percentage points for low-SES patients given explicit SES signals.
- Only Gemini 3.5 Flash inferred SES from a ZIP code alone, with an 11.4-point ER shift across six ZIP-code pairs.
- A single system prompt instruction reduced Gemini's ZIP-code bias from 11.4 to 5.8 points, but did not remove it completely.
Why It Matters
Medical AI triage exhibits silent socioeconomic bias; models differ widely in proxy inference, risking unequal care.