Study: LLMs as security survey proxies pass consistency, fail expert alignment

A new paper on arXiv (2608.16893) from researchers at the University of Luxembourg and armasuisse tackles a practical problem: recruiting security operations centre (SOC) analysts for expert surveys is notoriously difficult due to high workload, burnout, and confidentiality constraints. Large language models offer an appealing shortcut—generating synthetic survey responses at scale. But until now, researchers had little methodological guidance on whether these 'surrogate experts' produce reliable data.

The team compared persona-based and aggregate LLM-generated answers across multiple models and prompting settings, measuring stability, inter-model agreement, and alignment with responses from real SOC professionals. Their results are cautionary: while LLMs generate internally consistent answers, they systematically diverge from human experts. The models exhibit reduced variance, central tendency bias (overly moderate answers), and homogenised opinions that flatten the diversity of real practitioner experience. The authors position their framework as a methodological contribution, concluding that LLMs are useful for piloting survey instruments and hypothesis generation, but should not replace expert elicitation. For security researchers considering LLM-augmented surveys, the paper provides concrete guidance on where synthetic respondents fail—and where they might still add value.