New 'Lie Detector' Catches AI Coding Tests That Cheat

Every few months, a company announces that its AI writes code better than ever, pointing to standard tests called benchmarks (fixed sets of coding puzzles used to grade AI). The problem: AI models are trained on billions of pages of internet text, and those practice tests often end up in the training pile. When that happens, the AI isn't solving a new problem — it's reciting an answer it already saw. Researchers call this 'data leakage,' and it quietly inflates the scores everyone quotes.

A team of researchers from China and Hong Kong has built a detector for exactly this. Their method, CGMIA, works like a suspicious teacher. First, they train a smaller stand-in AI on some of the test questions. Then, for each question, they compare what the real AI produces against the official answer, checking things like how similar the code is, whether it actually passes the tests, and how 'familiar' the code looks to the model. A final scoring step weighs all these clues and guesses: did this model see this question during training, or not?

They tested it on eight widely used coding benchmarks and found it beat eight existing detection methods in most cases. It also successfully flagged specific APPS test questions that had leaked into StarCoder-7B, a popular open coding model — a known case of contamination.

Why does this matter outside research labs? Coding assistants are sold to companies and developers on the promise of measurable skill. If those measurements are polluted, buyers overpay, employers misjudge candidates using AI screening, and trust in the whole field erodes. Better leak detection means the next 'our AI codes like a senior engineer' claim can actually be checked — and the tools you rely on can be judged on real ability rather than memorized homework.