Code-MUE: New method measures LLM uncertainty via execution graphs

Code LLMs like GPT-4 and Claude are increasingly used for code generation, but their stochastic nature means they sometimes produce confident-looking but incorrect outputs—a critical risk in software engineering. Existing uncertainty estimation methods fall short: white-box and grey-box approaches require model access unavailable for closed-source APIs, while black-box text metrics conflate syntactic similarity with semantic correctness. To bridge this syntax-semantics gap, a team of researchers from multiple institutions introduces Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs.

Code-MUE generates multiple candidate code solutions for a given prompt, then executes them to observe runtime behavior. It builds a graph representing how code components interact during execution and calculates the Von Neumann entropy of the resulting solution space—a quantum-inspired metric that captures global semantic diversity. The core insight: higher entropy means more semantic spread, indicating lower confidence. In large-scale tests across eight state-of-the-art LLMs, Code-MUE achieved a Spearman correlation of up to -0.98 with functional correctness, far surpassing lexical and embedding-based baselines. This allows developers to automatically flag uncertain predictions for human review, enabling selective prediction workflows that improve reliability in production. The paper will appear at ISSTA 2026.