Doctors Want AI to Explain Your Hospital Discharge — It's Not Ready

When you leave the hospital, you get discharge instructions: which pills to take, when to come back, what warning signs to watch for. Studies show many people misunderstand them. So could an AI chatbot do the explaining? Researchers created DischargeBench, a simulated test. One AI acts as the hospital educator. A second AI plays a "virtual patient" with a specific personality, education level, health literacy and memory. A third AI keeps the fake patient realistic so the teacher isn't graded unfairly. They ran 477 real cases covering 24 disease categories.

The headline result is a warning about averages. Overall, the AI educators scored decently on conversation quality, covering the required topics, patient comprehension and staying factually consistent with the source document. But those averages hid big differences. When the virtual patient had low health literacy or poor recall, the AI skipped important topics, the patient understood less, and the AI's statements drifted further from the actual medical notes. In other words, the patients who need the most help got the weakest explanations.

This matters because most AI health tests grade the text the AI produces — is it accurate, is it well written — not whether a real human actually understood it. The authors argue that's the wrong yardstick. For a teaching task, understanding is the whole point. Hospitals are already rolling out AI tools for notes, messages and patient instructions, so how we measure them decides whether those tools help or quietly fail the people most at risk.

The catch: this is a simulation, not real patients, and part of the grading was done by AI judges rather than only doctors. So it's a red flag, not proof. Still, the practical takeaway is simple. If AI ever explains your care, bring a person you trust, ask questions, and never toss the paper instructions.