A Black-Box Procedure for LLM Confidence in Critical Applications

Engineering leader Jadair has published a practical, black-box procedure on LessWrong for assessing the confidence of Large Language Models (LLMs) in critical applications. The method addresses a core problem: frontier models like GPT-4, Claude 3, and Llama 3 are often overconfident, providing high-confidence answers that can be 30% wrong—a catastrophic risk in fields like aerospace or medicine. The procedure requires no internal model data, instead using three external checks: analyzing Google search result counts to estimate training data density for a topic, repeating questions across independent sessions to quantify answer stability, and asking related questions with web search disabled to identify knowledge gaps.

In an exploratory investigation using 320 queries across 8 topics and testing 3 different LLMs over 4 independent runs, this simple method predicted model accuracy within 2%, averaging less than 0.5% error. The work builds on existing research, including studies on training data density (LMD3) and the 'consistency hypothesis,' which states that answer consistency predicts accuracy. Jadair emphasizes this is hypothesis-generating work, not conclusive, but it provides a crucial, immediate tool for professionals. With industry transparency declining per the 2025 Stanford Foundation Model Transparency Index, this user-side method empowers teams to implement a risk-informed framework for AI deployment where model-provided confidence scores are unreliable or absent.