Human Misperception of Generative-AI Alignment: A Laboratory Experiment
People assume AI thinks like them, but lab experiments show significant misalignment in economic choices.
A new study by economists Kevin He, Ran Shorrer, and Mengjia Xia reveals a critical blind spot in how people perceive AI. Published on arXiv, the paper 'Human Misperception of Generative-AI Alignment: A Laboratory Experiment' used incentivized tests where participants made economic decisions for themselves and predicted what choices a generative AI would make on a human's behalf. The problems spanned domains like risk assessment, time preference, and strategic interaction.
The results were consistent and striking: in every single problem, the average human prediction about the AI's choice was significantly closer to the average human choice than to the AI's actual choice. This indicates a systematic overestimation of alignment. At an individual level, a subject's prediction about the AI was highly correlated with their own personal choice, suggesting people project their own reasoning onto the AI.
The researchers also modeled the implications of this misperception, highlighting a practical risk. When users assume an AI agent is more aligned with their values than it truly is, they may grant it greater autonomy or trust in sensitive areas—like financial planning or negotiations—without adequate oversight. This gap between perception and reality is a foundational challenge for deploying AI assistants in real-world decision-making scenarios.
- Study found humans overestimate AI alignment in every tested economic decision domain (risk, time, social preference).
- Individual predictions about AI choices were highly correlated with the predictor's own personal choices, showing a projection bias.
- The misperception gap has real-world implications for trust and autonomy granted to AI agents in finance and strategy.
Why It Matters
Overestimating AI alignment could lead to misplaced trust in AI agents for critical financial, strategic, or personal decisions.