LLM UX micro-simulation protocol validates feedback without real users

Early-stage product teams rarely have users, time, or budget for repeated UX studies—but they still need reliable signals to iterate safely. In a new paper on arXiv (2608.13563), researcher Alexandre Cristovão Maiorano proposes an artifact-first protocol that uses LLM-driven UX micro-simulations to generate structured customer-experience feedback, including walkthrough steps, friction points, and micro-survey signals, all derived from versioned prompts, personas, tasks, and UI snapshots.

To validate the simulated feedback, Maiorano leverages multiple public proxy corpora—app reviews, support tweets, and open-source issues—since public usability datasets with task outcomes are scarce. The protocol introduces two alignment metrics: top-k Jaccard and distributional weighted-Jaccard (W), testing lexical, TF-IDF, and multilingual embedding baselines across six proxy datasets. Embedding-based alignment consistently beat lexical baselines on primary app-review and support-tweet proxies (e.g., W=0.128 vs 0.000 on Gojek), while top-k Jaccard was shown to overstate alignment at larger k. The paper also ablates four agent strategies—single-pass, best-of-N, hybrid, and a score-then-select judge—and reports bootstrap confidence intervals revealing that the embedding W point estimate is systematically unstable at the chosen subsample size. A failure-mode analysis documents grounding and fabrication proxies, with outputs flagged as fabricated by an adversarial judge. The result is reproducible tables and figures from versioned run artifacts, enabling iterative prompt and taxonomy refinement before expensive paid-model calibration.