AI Codes Better When Its Notes to Itself Are Actually Right

When AI tools like ChatGPT write code, they often leave behind comments — short plain-English notes explaining what the code is supposed to do. Those notes stick around and become part of what the AI reads while writing the next chunk of code. A team of researchers asked a simple question: do those notes actually help, or are they just decoration? It turns out the answer is more interesting than expected.

They ran two kinds of tests. First, on a standard coding test set called LiveCodeBench, they found that neither how many comments the AI wrote nor what those comments were broadly about reliably predicted success. Then they did something clever: they took notes written by strong AI models and fed them to weaker ones. Notes that came from solutions that passed the tests raised the weaker models' success rate by 17.2% on average. Notes from failed attempts gave no reliable improvement. And notes written for a completely different problem actually made performance 20.8% worse.

The punchline is that it is not the format of the notes that matters — it is the correct solution content hidden inside them. When the researchers tried to get weaker models to produce that same benefit just by being instructed or prompted better, almost none could do it. The best case recovered only 24% of the gain. That means comments carry real problem-solving information that ordinary instructions simply cannot pull out of a model.

So what does this mean in practice? If you use an AI coding assistant at work, the quality of the information you feed it matters enormously. Give it accurate context, correct notes, or examples of code that actually works, and it performs noticeably better. Feed it outdated, wrong, or irrelevant material and you will actively make things worse — like handing a colleague a sticky note about a completely different project and expecting them to build the right thing.