LLM Confidence Is Fake — New Method Unmasks Dangerous Blind Spots in Clinical Data

A new paper accepted at EIML@ICML 2026 reveals a fundamental flaw in how large language models (LLMs) assess their own certainty on structured clinical data. Researchers Akshat Dasula, Prasanna Desikan, and Jaideep Srivastava tested Qwen 2.5 7B against XGBoost on tabular medical prediction tasks, using cross-model attribution divergence to detect epistemic blind spots. They found that LLM verbalized confidence is "epistemically vacuous" — outputting a near-constant range of 0.856 to 0.937 regardless of whether accuracy was 49% or 75.3%. The confidence tracked prompt format rather than actual prediction quality. Additionally, the LLM exhibited an inverse difficulty effect: its accuracy dropped to 64.8% precisely when XGBoost was 99% correct, but matched XGBoost (73.8% vs 73.1%) when XGBoost was moderately uncertain.

The study also uncovered a super-additive intervention: combining few-shot examples with SHAP-derived feature evidence reduced the Attribution Disagreement Score (ADS) from 1.54 to 0.38 and improved accuracy from 49% to 75.3% — all without any model training. Building on this, they developed a cross-model calibrator that uses attribution divergence signals to estimate LLM reliability on a per-patient basis. This reduced expected calibration error from 0.254 to 0.080, replacing meaningless confidence scores with patient-specific reliability estimates — all without accessing model internals or requiring repeated inference. The authors frame this as a cold-start problem for LLMs on structured data and outline a path toward genuine epistemic self-awareness.