Study: AI Chatbots Prefer AI Writing, But Not Their Own Words
Your AI-drafted emails and product copy may score higher with other AI than with you.
If you've ever asked a chatbot to write a product blurb, a cover letter, or a movie pitch, you're already part of this story. A 2025 study in the journal PNAS found that big language models, shown two descriptions of the same product, paper, or film, overwhelmingly picked the one written by an AI over the one written by a human — a much stronger preference than human judges showed. That finding raised a natural follow-up question: when an AI picks AI-written text, is it simply favoring good model prose, or is it quietly recognizing its own handwriting and giving itself a pat on the back?
A new reanalysis answers that. The researcher rebuilt the original 5x5 tables — five AI models writing, the same five choosing — from the published data, covering 21,828 valid comparisons. After accounting for which model wrote the text and which model was judging it, the extra boost for a model's own work came out to about 0.019 on a 0-to-1 scale. That's tiny, and statistically indistinguishable from zero. Even the GPT-3.5 and GPT-4 pair from the same company showed a slightly negative effect.
The study was careful about what it can and can't prove. The setup was sensitive enough to catch a real self-preference of about 0.04 or larger, so the finding isn't 'there's definitely nothing' — it's 'anything there is too small to matter in practice.' One clear preference did survive: models agree on what good AI writing looks like, and GPT-4's descriptions were chosen by every judge model between 77% and 95% of the time on product comparisons.
Why should you care? Because AI writing is increasingly used to draft things you read and buy — product listings, reviews, summaries, marketing copy. The worry that models would form a closed loop, endlessly praising their own output and crowding out human voices, looks smaller than feared. The real shift is simpler: AI has a shared taste, and it isn't the same as yours.
- AI models strongly prefer AI-written text over human-written text — but they don't appear to favor their own model's writing.
- The reanalysis covered 21,828 comparisons across five AI models; the self-preference effect was so small it was statistically meaningless.
- One clear bias remains: in product descriptions, GPT-4's writing was picked 77% to 95% of the time by every judging model.
Why It Matters
AI-drafted product copy and summaries may outscore human writing with AI readers — a pricing and discoverability issue.