LLM Urban Sims Fail Realistic Mobility: Study Tests AgentSociety & CitySim
AgentSociety and CitySim generate plausible cities but can't mimic real human movement.
A new paper by Santos, Viana, and Silva introduces a rigorous validation framework for LLM-based urban simulators, testing whether these generative agents produce empirically realistic human mobility patterns—or just plausible stories. They evaluated AgentSociety and CitySim using real-world datasets from Greater Paris and Shanghai, applying metrics like mobility laws, temporal rhythms, network motifs, semantic activity transitions, and behavioral profiles.
The results reveal a stark gap: while the simulators captured high-level semantic activity distributions, they failed to reproduce core spatial and temporal constraints—including trip-length distributions, origin-destination flows, dwell times, and transition dynamics. Interestingly, realistic mobility diversity was unstable across default prompting configurations and required explicit profile-aware initialization. To help improve reproducibility, the authors also contribute open LLM-driven infrastructure for map generation, observability-enhanced simulation, and traffic simulation.
- CitySim and AgentSociety failed to replicate trip-length distributions and origin-destination flows from real Paris and Shanghai data.
- Only high-level semantic activity distributions were plausible; fine-grained temporal rhythms and network motifs were inaccurate.
- The study provides open infrastructure for scalable map generation, simulation observation, and metric computation to standardize evaluation.
Why It Matters
Urban planners and AI researchers can't trust LLM-based simulations without empirical validation—this framework shows how.