Why AI Aces Math but Flunks Everything Else: A New Theory

For years, people have explained AI's math skills with a simple idea: math is easy to check. A wrong answer can be caught. So training an AI on math is like handing it an answer key. Steven Byrnes, writing on the AI blog LessWrong, says that explanation doesn't hold up. Checking advanced math isn't done by a calculator — it's done by another AI. So the real question is why AI is good at judging math in the first place.

His answer: the text AI learns from. Almost every sentence in a published math paper is true — he puts the figure above 99%. AI mostly learns by copying patterns from the writing it reads. So if it reads overwhelmingly correct math, it produces mostly correct math. Code works the same way: most code shared online actually runs and does what it claims, even when it's messy.

Other fields are different. In large chunks of medical, social science, nutrition, and self-help writing, true findings sit next to confident nonsense with no labels. An AI trained mostly by imitation soaks up both and repeats both, sounding equally sure either way — just like a person who believes everything they read online. Cleaning that up takes far more hand-picked training material, similar to what companies already do for code.

Byrnes notes this matters for the big question of whether AI can improve itself: whether AI climbs toward superintelligence depends on whether its source material is like math papers (reliable) or like the general literature (a mess). For everyday users, the takeaway is simpler. AI's confidence tells you nothing about whether it's right. Lean on it for math, code, and clear-cut facts. Double-check it on health, money, and anything where the internet itself is unreliable.