LLM prompting recovers hidden PHI with 98.1% recall, beating de-identification tools

Researchers at Texas Children's Hospital and Baylor College of Medicine tested whether LLMs with in-context learning (ICL) can close a critical gap in medical de-identification: institutionally situated protected health information (PHI) like hospital abbreviations, building names, and internal codes that standard tools miss. They benchmarked eight LLMs against two purpose-built systems (Stanford TiDE, OpenMed PII) and pattern-based baselines on 100 annotated pediatric oncology notes containing 5,322 PHI spans, using three prompts of increasing specificity: HIPAA-aligned baseline, baseline plus missed institutional categories, and a version that also discouraged over-redacting clinical content.

The results were striking. The best single LLM prompt achieved F1=0.918±0.001 versus TiDE's 0.779, with advantages concentrated in contextual categories that pattern-matching systems inherently can't capture. Simply naming the missed categories in the prompt recovered 79% (48/61) of them, while anti-over-redaction instructions restored precision — resolving the classic precision–recall trade-off in one LLM call per note. Notably, no agentic or multi-agent ensemble beat calibrated single-pass prompting (best F1 0.906–0.907). The LLM outputs also flagged 414 candidate annotation gaps; human re-annotation confirmed 227 real PHI spans, and the final prompt reached recall=0.981 (F1=0.907±0.002). Authors conclude that well-calibrated ICL is a legitimate, adaptable alternative to purpose-built de-identification systems, with institution-specific prompt development as the primary adaptation strategy — despite higher compute costs, the approach enables auditing the reference standard itself.