AI Writes Code That Works — But Forgets What It Needs to Run

Researchers Bhanu Prakash Vangala and Tanu Malik tested AI coding tools — the kind that write software when you describe what you want in plain English. They gave three different AI agents 50 programming tasks across four languages. The code the AI produced usually worked when tested. But the instructions about what else the program needs in order to run — the supporting software, versions, and settings — were a mess. Think of it like a recipe that tastes great but never tells you which oven to use.

The numbers are striking. When two AI agents were given the exact same task, their lists of required software agreed only 7% of the time. Some lists were incomplete, some repeated the same item twice, and some included things nobody needed. Newer, bigger AI models did no better than older ones. That matters because it suggests this isn't a problem that gets fixed by simply making AI smarter or more recent.

Why should you care? Any company using AI to write code inherits this cleanup work. Developers call it the 'it works on my machine' problem — the code runs fine on one computer and breaks on another, sometimes days later in front of a customer. Fixing it burns engineering hours, delays product launches, and can create security gaps if the wrong software gets pulled in.

The catch: this study measures the problem; it doesn't solve it. The authors argue that AI companies should train and grade their models on this, not just on whether the code passes tests. Until that happens, treat AI-written code like a draft from a talented intern — someone still has to check the setup before it ships.