Study: We Can Barely Tell If AI Was Trained on Your Writing

When an AI predicts a sentence unusually easily, people assume that sentence was in its training data. That assumption sits behind a lot of privacy complaints and copyright lawsuits. But until now, almost every test of it had to guess which sentences the AI actually saw. This paper removes the guessing by using two AI families, OLMo-2 and Pythia, that publish exactly what they were trained on, plus a public index showing how many times each sentence appears.

With the guessing gone, the result is deflating. Across five models ranging from 1 billion to 13 billion parameters (roughly, small to mid-sized AI brains), ordinary text left almost no detectable trace. The researcher scored this near -0.08 on a scale where -1 means a perfect link and 0 means none. In plain terms: essentially nothing. Only when a sentence was duplicated more than a thousand times did the signal become strong — and those sentences are the famous ones. The two training sets agreed on which ones they were, which means the AI isn't revealing secret memory. It's just good at well-known text.

The paper also shows how fake evidence gets manufactured. A common trick for testing is to take a sentence the AI supposedly memorized and change one word. The AI does prefer the original — but by the same amount whether the original appeared once or a hundred times. So it's rewarding the author's word choice, not recalling anything. Swapping in comparison sentences written in a different style pushed a detector's accuracy from 0.83 to 0.94, where 0.5 is a coin flip and 1.0 is perfect. That's a big jump caused entirely by how the test was built, not by any real memory.

The author released the sentence banks, duplication counts, and code so others can check the work. The takeaway isn't that AI never memorizes things. It's that the tools we use to prove it mostly measure fame, fluency, and test design — three things that look a lot like guilt but aren't.