New Paper Explains Why AI Chatbots Confidently Make Things Up

A mathematician named Adnan Aboulalaâ has published a long explainer paper that does something useful: it gathers, in one place, the probability theory that secretly runs modern AI. His core point is simple. A chatbot like ChatGPT is not looking anything up or reasoning like a person. It is playing an endless game of "guess the next word," using odds learned from enormous amounts of text. Every answer you get is the result of thousands of tiny coin flips, stacked one after another.

The paper describes training as a game of maximizing likelihood — basically, tuning the machine so that real sentences become more probable. Generation is then just rolling those loaded dice, one word at a time. Here is the crucial part: the math uses something called Kullback-Leibler divergence, which measures how far off a guess is. That measure is lopsided. It punishes a model far more for missing an obvious word than for saying something merely plausible. So the machine is trained to sound likely — not to be true.

That single asymmetry explains a lot of what frustrates you about AI. When a chatbot invents a fake court case, a bogus statistic, or a citation that doesn't exist, it isn't broken. It is doing precisely what it was built to do: produce the most statistically plausible next sentence. The paper draws a clean line between statistical plausibility and truth, and shows they are not the same thing. Fluent and confident, it turns out, are cheap. Accurate is expensive.

As a contrast, the paper also walks through diffusion models — the technology behind image generators like DALL-E and Midjourney. These don't predict one pixel at a time. Instead, they start with pure random noise and gradually erase it, step by step, until a clear picture emerges. Different math, same underlying goal: turn randomness into something that looks real. The takeaway for you is not technical at all. It is a rule of thumb: the smoother and more confident an AI answer sounds, the more you should check the facts.