AAAI study: Synthetic LLM agents fail legitimacy test in democracy

As large language models grow more persuasive, researchers and institutions are increasingly deploying synthetic agents—LLM-powered stand-ins—in place of human participants across user testing, market research, surveys, and qualitative studies. But a new paper from Aditya Nayak, Aditi Vashistha, Alissa Centivany, and Aakash Gautam, accepted to the 9th AAAI Conference on AI, Ethics, and Society (AIES 2026), warns that this trend has crossed into dangerous territory: experimental implementations in policy consultation, jury deliberation, and humanitarian diplomacy. The authors argue that participation is not merely about providing information or reaching consensus—it is a legitimizing condition for democratic institutions. Treating synthetic agents as human substitutes, they claim, raises serious political, representational, and ethical concerns that 'appearing legitimate' cannot resolve.

To dissect these issues, the researchers apply a Participatory Design lens—rooted in probing, priming, understanding, and generating—to three case studies spanning representational scales: local policymaking, enterprise jury deliberation, and global diplomacy. They argue that legitimacy and personhood are integral and mutually constitutive, meaning that replacing human voices with AI agents breaks the foundational representational bond that gives institutions their authority. The paper identifies ethical, representational, and methodological risks, and concludes by proposing soft and hard boundaries for designing oversight on LLMs and synthetic agents in representational processes. These boundaries aim to help policymakers and technologists distinguish between AI tools that support human participation and those that quietly substitute for it, ensuring that synthetic agents never become a shortcut around democratic accountability.