AI Can't Fake a Nation's Opinions — It Just Guesses Stereotypes
A new study warns AI-simulated polls could mislead companies and governments
Silicon sampling uses large language models to simulate survey respondents — and a study tested it against the European Social Survey Round 11 (30 countries, 42 items) using two open-weight LLMs. Aggregate recovery was moderate and uneven across items. Adding the country name to a three-variable demographic backstory raised the median per-item correlation between simulated and observed country means from -0.03 to 0.52, while richer profiles added no consistent gain — the country label acted as a country-level assumption that respondent detail did not revise. Individual-level recovery remained negligible in every condition. And an average of neighboring countries, using no LLM, recovered country levels more accurately than every model condition and ranked them about as well.
- AI survey stand-ins only look accurate when told which country they're imitating — so they're echoing stereotypes, not real opinions
- Naming the country lifted the AI's match with real national averages from about zero to 0.52, while adding job, age and education changed nothing
- Simply averaging neighboring countries' real survey results beat the AI in every test, and the AI never predicted individual people's answers
Why It Matters
If AI fakes public opinion, bad decisions on policy, products and campaigns follow — while real people go unheard.