Post-training turns LLMs into effective medical coders
Prompting-only tests massively underestimate LLM coding ability.
A team of researchers from Northwestern University and others conducted the first controlled empirical study on post-training generative LLMs for automated ICD coding—a critical task for medical billing, epidemiology, and clinical decision support. While prior work dismissed LLMs as weak medical coders, those conclusions were drawn from inference-time setups like prompting, retrieval, or tool use. This study shows that such evaluations drastically understate what LLMs can achieve when properly adapted. Using a common protocol and metric set, they compared discriminative baselines with LLMs under prompting, supervised fine-tuning (SFT), and reinforcement learning.
The results are clear: SFT provides the main capability jump for code-set prediction, while GRPO (a variant of GRPO) further refines performance beyond SFT. The team also introduced PHI, a diagnostic curriculum that extends GRPO to specifically improve recall on missed-code cases, boosting macro-level performance. Their findings suggest the bottleneck isn't the generative formulation itself, but how the model is optimized for full-taxonomy recall. The paper, code, and checkpoints are publicly available, offering a roadmap for deploying LLMs in production medical coding systems.
- Prompting-only evaluation severely underestimates LLM potential for ICD coding compared to post-training approaches.
- Supervised fine-tuning (SFT) delivers the largest performance improvement, with reinforcement learning (GRPO) adding further gains.
- A new curriculum called PHI extends GRPO to target missed-code cases, improving macro-level recall.
Why It Matters
Enables accurate, automated medical coding at scale, reducing billing errors and supporting clinical decision support.