New Tool Makes AI-Written Code Provably Safe Before It Runs

AI coding assistants now write software faster than any human team can read it. That speed is wonderful until something breaks: a payment script that transfers the wrong amount, a robot arm that swings too far, or a chip instruction that overheats a machine. Today's safety nets — fuzz testing (throwing random inputs at code to see what breaks) and static analysis (scanning for known bug patterns) — catch plenty, but they can never check every possible case.

A team of researchers from the University of Wisconsin-Madison and collaborators built a system called MAGS to close that gap. Their idea borrows from formal verification (using math to prove, with certainty, that code does exactly what it's supposed to). Traditionally that requires a human expert weeks of painstaking proof-writing. MAGS splits the job among several AI agents: they take safety rules a human has reviewed and locked in, translate the AI-generated code into Dafny (a language built for mathematical proof), let the checker flag violations, repair them, and then compile the verified code back into something that actually runs.

They tested it on 220 pieces of real software: 100 CUDA kernels (the low-level code that runs on graphics chips), 100 terminal scripts (the command-line automation that runs servers), and 20 robotic-arm tasks. Every one of the 220 produced a program with meaningful, machine-checked safety guarantees against the locked specifications.

The honest limitation is the specification itself. MAGS only proves the code matches the rules humans wrote down. If those rules miss something important, the code can still be wrong in ways the math never catches — and the researchers say they saw exactly that in independent tests. It's a strong seatbelt, not a self-driving car. And it's a research paper, not a product you can install today.