AI Stand-Ins for Human Surveys? New Study Says Not So Fast

Imagine a survey where you choose between a cheap phone with a short battery life and an expensive one with a great camera. That's a conjoint experiment — a popular way to learn what people really value by presenting trade-offs. But recruiting thousands of humans is slow and pricey, so researchers wondered: could AI chatbots stand in for people and do the job faster and cheaper?

This new paper put that idea to the test. The authors took several published conjoint studies that used real human participants, then repeated them using synthetic agents — essentially AI personas meant to represent different kinds of people. They compared the AI answers to the original human data across three dimensions: the pattern of choices, the statistical estimates, and how stable the results were.

The results were mixed, and not in a reassuring way. In some studies, the AI reproduced the overall picture well enough — like matching the general shape of a chart or getting the direction of an effect right. But when researchers looked at finer details or repeated the exercise, the AI often drifted from human behavior. A sign of agreement on the surface wasn't enough to confirm deeper accuracy.

The authors' conclusion is blunt: using AI to replace human survey participants isn't ready for prime time. The reliability of synthetic people depends on what claim you're making — it might be fine for getting a rough sense of public opinion, but not for measuring nuanced preferences. The researchers urge caution before treating AI as a cheap shortcut for understanding what people truly want.