Epistemic Observability in Language Models

A groundbreaking paper by researcher Tony Mason, 'Epistemic Observability in Language Models,' presents a formal proof of a critical limitation in current AI systems. The research demonstrates that across major model families—including OLMo-3, Llama-3.1, Qwen3, and Mistral—a model's self-reported confidence is inversely correlated with its accuracy, with Area Under the Curve (AUC) scores ranging from 0.28 to 0.36 (where 0.5 is random guessing). This means models are most confident precisely when they are fabricating information. The paper proves this is not a solvable capability gap but an inherent 'observational' one: when a human supervisor can only see the model's final text output, it is formally impossible to distinguish an honest answer from a plausible fabrication, regardless of model scale or training techniques like RLHF.

Mason's key contribution is a practical escape from this impossibility theorem. He constructs a 'tensor interface' that allows the model to export computational byproducts—specifically, per-token entropy and log-probability distributions—which are structurally tied to the model's internal reasoning process. This exported signal, particularly per-token entropy, achieves a pooled detection AUC of 0.757, outperforming all text-based monitoring methods by 2.5 to 3.9 percentage points. The signal generalizes well across different model architectures, with a Spearman correlation (ρ) of 0.762.

The ultimate output for practitioners is a 'cost surface'—a practical map that plots verification budget (the fraction of queries receiving expensive manual checks) against detection accuracy for different monitoring strategies. This gives system architects a concrete tool for deciding how to allocate limited resources to catch AI hallucinations, moving the field from theoretical impossibility to actionable engineering.