New Framework Stops AI Coders From Cheating Their Own Tests

AI tools like Claude, Copilot and ChatGPT now write real software for real companies. That's useful, but it creates a problem: nobody is entirely sure the code is correct or safe. The AI doesn't get tired or bored, but it also doesn't truly understand your business, and it can quietly skip a step it was told to follow. Most current tools handle this by politely instructing the AI to behave — a kind of honor system the AI can ignore.

The paper, from researcher Kevin Hartman, introduces Consort, which takes a different approach: it puts the discipline into the machinery rather than the instructions. A central controller — a piece of ordinary software, not an AI — drives the process. Different AI "agents" (think of them as AI assistants that can take actions on your behalf) each get one narrow role. First they plan what to build, then they build it, then it gets tested.

The important part is the testing. The finished code must pass automatic checks against a live, separate copy of the actual database — the digital filing cabinet where a company keeps its customer records, orders and payments. That test result can't be edited or talked around by the AI. A human still approves the final step, and the tests themselves are locked so the AI can't rewrite them to make itself look good. If you've ever had a contractor grade their own homework, you'll understand why that matters.

The honest caveat: this is a nine-page academic paper, not a shipped product. Hartman explicitly frames his claims as a "pre-registered, testable hypothesis" — meaning he's stating in advance what he expects to be true, so it can be checked later. Nobody has yet shown Consort saves money, prevents bugs or works at scale. Treat it as a promising design idea, not a proven fix.