Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues

A team of researchers including Eric Gong, Nathan Sanders, and security expert Bruce Schneier has published a groundbreaking study on arXiv demonstrating a new, more effective method for using Large Language Models (LLMs) in political polling. Their framework bypasses the traditional approach of simulating thousands of individual AI respondents, which is computationally expensive and inconsistent. Instead, they directly prompt an LLM (like GPT-4 or Claude) to predict the overall distribution of responses to a multiple-choice polling question. This single-query method was tested against the high-quality Cooperative Election Study of the US population.

The results show the new distributional prompting framework consistently outperforms the individual simulation method in accuracy while reducing computational costs by approximately 90%. Crucially, the study found that the performance of this AI polling method varies much more systematically and predictably across different demographic groups and question topics. This predictability allows pollsters to better anticipate the model's reliability before running a query, addressing a major concern about "black box" AI outputs. The work highlights LLMs' potential to augment traditional surveys, which are becoming less tractable due to rising costs, non-response bias, and declining demographic coverage.