AI Pretending to Be Your Customer Works Worse Than Just Asking

Marketers increasingly ask AI chatbots to pretend to be customers — a 34-year-old mom in Ohio, a budget-conscious student — and then judge whether a headline or ad will work. The idea is that detailed fake people make better predictions than a generic guess. A new study put that idea to the test using the Upworthy Research Archive: thousands of real headlines that were shown to real people, with real click counts recorded.

The researcher compared two approaches. One built a panel of ten AI personas grounded in the real audience's demographics. The other simply asked the model how likely a typical reader was to click, with no role-playing at all. The plain version won clearly: it correctly picked the better headline 49.2% of the time, versus 34.6% for the persona panel. The finding held across three separate data splits, different ways of phrasing the prompt, and several AI models, including Google's Gemini tiers and OpenAI's GPT-4.1.

Why would fake people hurt? The researcher's explanation is simple. Asking a model directly taps a decent gut sense of what the general public likes. Forcing it to act out a specific character adds invented details, personal quirks, and noise that pull the answer off target. The personas were not just weak — they were worse than using no personas at all.

The catch: this measures what makes a whole audience click, not what appeals to a specific niche. If you are targeting a narrow group, personas might still have a place — this study just does not show it. There is also a hidden snag for anyone testing ads this way. Most A/B tests never produce a clear winner, so only 399 of the thousands of tests qualified as reliable enough to judge. In other words, the ground truth itself is fuzzy, which makes any AI prediction tool harder to trust than it looks.