New survey: Multimodal LLMs need uncertainty-aware decision making

A new survey paper on arXiv (2608.17084) from researchers Abderrahmene Boudiaf, Irfan Hussain, and Sajid Javed argues that multimodal large language models (MLLMs) need a fundamental shift in how uncertainty is evaluated. Titled "Uncertainty-Aware Decision Making in Multimodal Large Language Models," the work organizes the field around a decision-centered framework: uncertainty sources produce observable signals, signals must be calibrated or controlled for risk, and calibrated uncertainty should determine the system's action. This moves beyond treating uncertainty as a single confidence number, instead asking whether uncertainty improves behavior when visual, textual, acoustic, temporal, chart, or embodied evidence is insufficient, conflicting, or shifted.

The survey systematically reviews a wide range of techniques, including token and logit uncertainty, semantic disagreement, perturbation instability, grounding and attribution scores, verbalized confidence, verifier/judge scores, conformal prediction, selective answering, abstention, clarification, retrieval, self-checking, and escalation. It distinguishes itself from text-only uncertainty surveys, broad MLLM surveys, hallucination-focused surveys, and safety reviews. The authors conclude with open problems: source-aware decomposition, action-aware benchmarks, calibration under shift, black-box uncertainty estimation, broader modality coverage, reproducible reporting, and human-centered uncertainty communication. The key takeaway: a fluent answer may conceal a perceptual error, weak grounding, or an unanswerable question—so MLLMs need robust mechanisms to know when to abstain, clarify, or escalate.