MiroBench reveals LLMs fail to mimic real Reddit discussions with 4,292 threads

LLM agents are increasingly used to simulate human interactions — from customer support to social science studies — but until now there was no systematic way to measure whether those simulations actually mirror real behavior. A new paper from researchers including Yaoning Yu introduces MiroBench, a benchmark that takes a concrete first step by focusing on Reddit discussions. Reddit provides public, topic-grounded, multi-party interactions where people share experiences, debate, seek advice, and express emotion. MiroBench uses 4,292 real Reddit threads as ground truth and applies statistical tests to compare generated conversations against real ones across four key aspects: repetition and semantic uniformity, narrative content, toxicity and aggression, and structural complexity.

Experiments spanning five models (including GPT-4, Claude, and Llama variants) across five domains (e.g., tech, advice, politics) revealed a clear finding: current LLM-based simulators remain distributionally mismatched with real Reddit threads. They struggle to replicate the nuanced patterns of human conversation, such as the natural ebb of topic shifts, the specific blend of toxicity and politeness, and the structural flow of multi-party replies. A lightweight prompt-based improvement procedure was tested but provided only limited gains. MiroBench thus offers a much-needed concrete benchmark for measuring, diagnosing, and improving realism in agentic social simulation — a critical capability as AI agents move into roles that require believable human-like interaction.