Fine-tuned LLMs discover 1 in 5 UK police logs show mental health issues

Researchers Sam Relins and Daniel Birks adapted a fine-tuned LLM classification pipeline—originally trained on open-source US police data—to estimate the prevalence of four vulnerability indicators (mental ill health, substance misuse, alcohol dependence, and homelessness) in nearly 3,000 de-identified UK police incident logs. The pipeline combined repeated model inference, label aggregation, structured human review, and statistical correction, running entirely on a locally hosted open-weight LLM to meet secure police environments. Results showed mental ill health indicators in roughly one in five incidents, with lower prevalence for the other categories.

However, naive deployment proved unreliable: single-pass classifications were unstable, and aggregated outputs systematically over-assigned indicators relative to human judgment. Correcting these biases required substantial human input and statistical adjustment, leaving considerable uncertainty. At the population level, defensible estimates were achievable but resource-intensive; at the individual level, errors remained frequent and unpredictable, limiting suitability for operational decisions. The study highlights both the potential and constraints of LLM-based measurement in high-stakes applied settings.