RealUserSim grounds AI agent evaluations in real user data, boosting match rate from 24% to 45%

Current LLM-based user simulators suffer from two critical flaws: a Formalism Ceiling that achieves only 6–8% style match against real users, and Directive Amplification where hand-crafted instructions push simulated users into unnatural behavioral extremes that vary wildly across models. This renders agent benchmarks unreliable—simulated users are poor proxies for real humans. RealUserSim, developed by researchers including Ming Zhu and Silvio Savarese from Salesforce AI, solves this by grounding simulation in real behavioral data. Using over 14,000 authentic human-LLM conversations from the WildChat dataset, they extract 7,275 executable behavioral profiles. These profiles capture real user patterns—including inconsistencies, ambiguous requests, and context shifts—that unconstrained LLMs or rigid scripts cannot replicate.

On a new fidelity benchmark called PT3 (600 conversations across 71+ domains with anti-leakage controls), RealUserSim raises behavioral match rate from 24.2% to 45.3% across five key dimensions. When used to evaluate agents on the TauBench suite with six different simulator models, grounded simulation acts as a realistic stress test: it surfaces three distinct failure mechanisms that cooperative simulators completely miss, leading to a mean task success degradation of 3.2–3.5%. This proves that existing benchmarks inflate agent performance by using unrealistically compliant testers. RealUserSim offers the first viable path toward trustworthy agent evaluation by bridging the gap between lab benchmarks and real-world deployment conditions.