AI Struggles with Rare Disease Decisions: It Shares Care, Not Saves Lives
This could mean hospitals using AI might spread resources too thin in life-or-death situations.
A team of researchers tested 11 top AI systems on tough medical cases involving rare diseases. They presented the AIs with 208 scenarios where doctors must make heartbreaking choices between patients because resources are limited. Instead of picking the most critical case, the AI consistently chose to split resources equally—even if one patient was far sicker.
The study suggests this isn't just a quirk of AI. The systems showed a strong bias toward 'justice'—treating everyone the same—regardless of how sick someone was. In one test, the AI only changed its mind when the decision was framed as being made by a doctor or patient directly. This hints that AI might be copying the institutional priorities of hospitals or health systems rather than following medical urgency.
The findings raise alarms for real-world use. Imagine an AI helping a hospital decide which rare-disease patient gets the last bed in the ICU. If the AI insists on fairness over need, the patient who could have been saved might not make it. Doctors already face these agonizing calls, but AI is supposed to help, not repeat old mistakes—or worse, ignore life-or-death differences.
This isn’t just about rare diseases. If AI defaults to 'equal' instead of 'effective,' it could affect how AI is used in other high-stakes decisions—from disaster response to school funding.
- AI helping doctors tended to split care equally rather than prioritizing the sickest patients in rare-disease cases.
- All 11 tested AI models showed this bias, ignoring clinical severity and focusing only on fairness.
- The AI’s behavior changed depending on who was framed as making the final call—doctors, patients, or a committee.
Why It Matters
AI in healthcare might make life-or-death decisions based on fairness instead of urgent need, risking worse outcomes for the most vulnerable patients.