Survey of 18 LLMs: Medical specialist models outperform in diagnosis
Specialist LLMs excel at diagnosis while general models lead in dialogue and decision support.
A comprehensive survey from researchers led by Qi Peng presents a dual-view framework for aligning clinical needs with AI capabilities, evaluating 18 state-of-the-art large language models (LLMs) on medical reasoning. The study introduces a five-level competency ladder derived from Miller's Pyramid, ranging from knowledge recall to dynamic case management, and maps deductive, inductive, and abductive reasoning patterns to common medical goals. To benchmark performance, the team created a dataset spanning all five reasoning levels and tested models including both general-purpose LLMs and medically specialized ones.
The results reveal a clear specialization: medical specialist models significantly outperform general models on diagnosis-centric tasks, while general-purpose LLMs show stronger performance in decision support and natural dialogue. The paper also identifies critical open challenges—such as hallucination, grounding errors, and data scarcity—that hinder deployment in real clinical settings. The authors argue that future progress depends on bridging the gap between reasoning benchmarks and actual clinical workflows, and they outline directions for building safer, more reliable systems that clinicians can trust.
- Survey uses a five-level competency scheme based on Miller's Pyramid to evaluate LLM medical reasoning.
- 18 state-of-the-art models tested; specialist models excel in diagnosis, general models in dialogue and decision support.
- Major challenges include hallucination, data limitations, and grounding issues that prevent real-world clinical adoption.
Why It Matters
Helps healthcare organizations select the right LLM for the task: diagnosis vs. patient communication.