New benchmark OGCaReBench reveals LLMs flunk rare clinical cases
GPT-5.2 scores only 56% on off-guideline medical questions without external help.
A team of 14 researchers from medical and AI institutions has released OGCaReBench, a rigorous benchmark designed to test large language models on rare clinical scenarios that fall outside established medical guidelines. While most medical LLMs are trained on common, guideline-focused knowledge and evaluated via multiple-choice questions, this benchmark uses real published case reports validated by medical experts. The free-form, open-ended questions require nuanced reasoning that memorized knowledge cannot handle. In tests, even the top-performing model (GPT-5.2) correctly answered only 56% of the benchmark questions, and specialized medical models lagged further at 42%.
However, when models were augmented with retrieved medical articles (a technique called retrieval-augmented generation), GPT-5.2's performance soared to 82%. This dramatic improvement underscores the limitations of relying on parametric memory alone for high-stakes medical decisions. The paper establishes a foundation for benchmarking LLMs in challenging clinical contexts and highlights the critical need for evidence-grounding in real-world healthcare AI applications.
- OGCaReBench uses rare clinical case reports validated by medical experts, not standard guideline questions.
- GPT-5.2 scored only 56% without retrieval; specialized medical models scored just 42%.
- Retrieval-augmented generation boosted GPT-5.2 to 82%, proving the value of evidence grounding.
Why It Matters
Shows that current medical LLMs cannot handle rare cases without external retrieval, critical for safe deployment.