DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views

A team from Stanford University and the Deliberative Democracy Lab has introduced DeliberationBench, a first-of-its-kind framework designed to measure and evaluate the persuasive influence of Large Language Models (LLMs) on human beliefs. The core innovation is its normative standard: instead of judging influence as simply 'good' or 'bad,' the benchmark uses data from real-world Deliberative Polls—structured democratic discussions where citizens become more informed—as a gold standard for beneficial opinion change. This provides a defensible, process-oriented measure of whether an AI's influence aligns with democratically legitimate outcomes.

The researchers put this benchmark to the test in a large-scale, preregistered experiment. They had 4,088 U.S. participants discuss 65 diverse policy proposals with six unnamed 'frontier' LLMs (like GPT-4 and Claude 3). They then compared the resulting opinion shifts to data from four prior in-person Deliberative Polls. The key finding was a positive correlation: the LLMs nudged participants' views in directions similar to those achieved through high-quality human deliberation. This suggests that, in aggregate, current top-tier AI models exert an influence that is epistemically desirable, helping users form more considered opinions.

Further analysis explored differential effects. The study examined how influence varied across topic areas, demographic subgroups, and between different AI models. While the overall effect was positive, this granular look is crucial for identifying potential biases or uneven impacts. The DeliberationBench framework is ultimately proposed as an evaluation and monitoring tool for developers and policymakers. Its goal is to help ensure that as LLMs become ubiquitous 'thought partners,' their influence preserves user autonomy and remains consistent with democratic values, not manipulation.