LLMs audit Spanish apps in 24 EU languages, reveal privacy gaps

A team of researchers led by Marcos Moran has demonstrated that large language models (LLMs) can perform privacy policy audits across all 24 official EU languages without language-specific adaptations. They assembled an evaluation corpus by translating two expert-annotated datasets (OPP-115 and MAPP) and validated translation fidelity with automated metrics and legal expert review. Their LLM-based classifier for identifying categories of personal data collection achieved stable cross-lingual performance, with macro-F1 scores between 0.91 and 0.94 across all languages.

Applying their method to a real-world audit of 2,611 Android apps from the Spanish Google Play Store, the researchers combined multilingual policy analysis with evaluations of privacy labels and runtime network traffic. They discovered a striking linguistic barrier: public-sector apps predominantly provided privacy policies only in Spanish, while popular commercial apps offered them in English. More critically, they found systematic discrepancies between what apps declared in their privacy policies and what they actually did—especially in public-sector apps. The study concludes that English-only privacy audits can systematically obfuscate transparency gaps in multilingual environments, making this LLM-based approach a crucial tool for regulators and watchdogs.