Stanford researchers reveal why LLMs struggle with ambiguity

Researchers from Stanford University (Nicolas Zucchet, Hyun Dong Lee, Scott Linderman) published a paper titled *'Language models suffer from a curse of ambiguity'* on arXiv, exposing a fundamental limitation in how large language models (LLMs) handle probabilistic outputs. The study argues that ambiguity in next-token distributions—where multiple plausible continuations exist—creates a 'curse' that degrades model learning efficiency. Through theoretical analysis and experiments on synthetic and real-world data, the team found that ambiguous distributions demand more computational capacity, larger embedding dimensions, and longer training times to fit accurately. Additionally, sampling noise is amplified in ambiguous settings, further degrading performance.

The paper provides a practical framework for evaluating when to trust LLM outputs by quantifying ambiguity’s impact on distribution fidelity. The researchers validated their findings on controlled synthetic tasks and observed identical patterns in models trained on real data. This work offers a new lens for understanding statistical limitations in LLMs and could guide improvements in architecture design, training protocols, or output calibration to mitigate ambiguity-driven failures.