AI Still Can't Read Medical Records Like a Doctor Can

Medical researchers spend enormous amounts of time reading patient records and pulling out the details that matter — tumor size, treatment history, side effects. It's slow, expensive, and only trained experts can do it well. So a team of researchers built an AI tool called Libretto, powered by the same kind of technology behind ChatGPT, to lend a hand. They co-designed it with seven cancer research teams and then asked each team to use it on real questions from their own work.

The results were humbling. Clinicians knew exactly where and how to spot complicated concepts in patient notes — they've done it for years. But in 12 of the 14 tasks, they couldn't get the AI to reproduce that instinct. Three problems kept showing up. First, doctors make constant judgment calls about whether a note is trustworthy, and the AI had no way to inherit those judgments. Second, steering the AI meant "vibe-coding" — describing what you want in plain English and hoping it lands — which turned out to be frustrating and imprecise. Third, the ways they could check the AI's work were too rigid for real research questions.

Here's why that matters beyond the lab. Medical records are the raw material for nearly every advance in cancer care, from spotting early warning signs to testing new drugs. If AI can't reliably do this reading, then either patients wait longer for discoveries, or hospitals pay for armies of human reviewers, or — the real danger — someone trusts an AI's output without catching the mistakes. The study's authors are clear that this isn't a reason to abandon the idea. It's a design problem: the tools need to fit how experts actually think, not the other way around.

One caveat before you draw big conclusions: this was a small study of seven research teams and 14 tasks, done with a custom-built tool rather than a commercial product. It's a snapshot of where things stand, not a verdict on what AI can do in medicine. But it's a useful reminder that "the AI can read documents" and "the AI can read your medical file correctly" are very different claims.