AI Wrote the Fix for a Bug That Randomly Broke AI Software

PyTorch is the software toolkit that most AI models are built with, and it comes with thousands of self-checks that run automatically every time anyone changes the code. One of those checks kept failing at random — passing most of the time, then failing for no obvious reason. Engineers call this a "flaky test," and it's a genuine headache: it wastes hours, and it trains people to ignore warnings that might matter.

The cause turned out to be a speed race with absurdly fine margins. PyTorch sometimes reorganizes numbers into tidier rows before multiplying them, but only if that makes things faster. To decide, it actually times both versions. At these sizes each timing run finishes in under a tenth of a millisecond — far too fast to measure reliably, so the decision came down to noise. When it chose to pad, the software quietly produced one extra small program for the graphics chip, and the test — which expects exactly one — threw up its hands. Under a busy, stressed machine, the wrong choice happened 6 times in 100 runs. Those six were exactly the six failures.

The fix is refreshingly simple: turn that optimization off for this one test. The test still checks what it was built to check, but the result no longer depends on a coin flip. Under the same heavy load, the test now passes 100 out of 100 times.

One more detail worth noticing: the patch was drafted with the help of an AI coding assistant, with a human verifying every measurement and test result by hand. That's increasingly how software gets fixed — AI does a first pass, a person checks the math.