AMD's AI Chips Have a Tiny Math Bug — and Engineers Just Caught It

A routine quality test inside PyTorch, the free software toolkit that powers most AI models, caught something odd: AMD's MI200 AI chips occasionally return slightly wrong numbers on one particular kind of calculation. Out of 56 values checked, one was off — by about 1% — which sounds trivial until you remember AI models run billions of these calculations in a row.

The strangest part is that the error comes and goes. When the chip runs this math, it picks between three internal methods, essentially three recipes for the same dish. Which recipe gets used depends on which one happens to look fastest at that moment. One of those three recipes, it turns out, is about ten times less accurate than it should be. Force that one, and the test fails every single time. Force either of the others, and it always passes. On a busy shared computer, the winner varies run to run — hence the flickering failures.

The engineering team had three options, and the two tempting ones were bad. They could relax the accuracy standard to make the failure disappear, but that would paper over a genuine chip bug and the same standard is shared with other hardware. They could force a specific recipe from within the test, but that would change settings for every other test running alongside it. So they took the honest route: disable this one test only on the MI200 chip, since it passes fine on newer AMD chips, and hand AMD a detailed bug report with a small program that reproduces the problem.

Why should you care? Because AI answers are only as reliable as the arithmetic underneath them, and small errors can quietly stack up. Such errors are usually caught. This fix also shows a healthy habit worth knowing about: the flaw was documented publicly and reported upstream rather than swept aside — and, notably, the fix itself was drafted with help from an AI coding assistant, then reviewed by a human engineer.