Study: AI 'Fake People' Still Can't Replace Real Human Surveys

Companies and researchers are exploring a big money-saving idea: instead of paying thousands of people to answer surveys, why not create AI-generated "personas" — fake but realistic-sounding people — and ask them what they think? A new study put that idea to a tough test in South Korea, running about 8,000 AI personas through questions about digital and AI service use, then comparing their answers to real responses from a national survey.

The results were not flattering. On average, the AI personas were off by 15 to 19 percentage points — a huge gap when you're trying to measure something like how many people use a specific app. They also made predictable mistakes: one AI model stereotyped older people, while another tended to agree with everything (a problem called acquiescence bias). In other words, the AI didn't just fail randomly; it failed in ways that could mislead anyone making decisions from this data.

The researchers tried to fix the problem by "calibrating" the AI — essentially teaching it using 30% of the real survey data. That cut the errors roughly in half, but it still wasn't nearly as accurate as simply using that same 30% of real responses directly. The only time the calibrated AI helped was when researchers had almost no real data at all — fewer than 100 real responses — or when they wanted to guess at groups that weren't in the real sample.

So what's the takeaway? Synthetic survey respondents are not a replacement for talking to actual humans. They might be useful for early brainstorming or exploring ideas when you have zero data, but for anything serious — tracking public opinion, planning a product launch, shaping policy — real surveys still win. The study's authors are blunt: AI personas are best seen as diagnostic tools, not substitutes. Until AI can truly mimic the complexity and inconsistency of real people, your gut feeling might be more reliable than a bot's answer.