LLM decoding: higher token probability doesn't guarantee correct answers

A new preprint by Johannes Zenn and Jonas Geiping (arXiv:2606.27359) systematically examines a core assumption behind many LLM decoding strategies: that shifting probability mass toward more likely outputs leads to more correct answers. The authors analyze the relationship between sequence probability and correctness at four distinct levels: across decoding methods (e.g., temperature scaling, top-k, beam search), across hyperparameters within a single method, across different prompt-answer pairs in a fixed dataset, and across multiple responses to the same prompt. The goal is to provide practical guidance for techniques like self-consistency and verifier-free self-improvement.

The results reveal a nuanced picture. Higher sequence probability is often predictive of correctness when comparing different prompt-answer pairs within a fixed dataset. However, this relationship breaks down when applied to decoding decisions: increasing sequence probability by tweaking hyperparameters or switching decoding methods does not reliably improve accuracy. Even more striking, sequence probability is a poor indicator of correctness for multiple responses to the same prompt. These findings clarify that while probability can serve as a useful heuristic in some contexts, it cannot be universally trusted, and they offer concrete boundaries for when decoding and self-improvement strategies can be expected to succeed.