New CARE framework reveals LLMs can't mimic online community attitudes

A new paper from researchers Nuan Wen and Xuezhe Ma at the University of Southern California introduces CARE (Community-Aware Reaction Evaluation), a framework designed to assess how well large language models (LLMs) simulate the nuanced, event-contingent reactions of real online communities. The work tackles a fundamental flaw in current LLM-as-proxy evaluations: they often reduce social identity to static labels, ignoring how groups actually respond to shifting real-world events. CARE replaces simplistic prompts with a granular analysis of "illocutionary tones"—the attitudinal subtext behind written reactions—validated through a human-AI collaborative loop.

When tested against authentic community responses to news, the diagnosis reveals a stark "realism gap." Simply steering an LLM with explicit community-focused prompts does not inherently improve simulation fidelity. Moreover, different frontier models exhibit divergent behavioral signatures, suggesting that current alignment strategies are insufficient for capturing the sociolinguistic dynamics of online groups. The findings have significant implications for computational social science, market research, and any application relying on LLMs to proxy human discourse—the models are still not fooling real communities.