Study finds LLMs up to 17% less accurate in non-English languages

A new paper from researchers Rafael da Silva and Jeff Eicher introduces the Cross-Lingual Comprehension Gap (CLCG), a metric for measuring how much a language model's answer quality drops when the same content and question appear in a language other than English. The study uses ParallelQA-18, a professionally human-translated parallel corpus, and evaluates five models from five laboratories on a stratified sample of 150 Wikipedia-style articles across 18 languages—including Portuguese as a high-resource baseline and 16 targets spanning Joshi resource classes 0 to 4. By holding content, question, reference answer, model, and evaluation unit constant, the within-item design isolates language as the only variable. The primary estimator compared English versus pooled target-language Token-F1 micro-means on higher-complexity open-ended questions, using article-cluster bootstrap intervals. The resulting pooled CLCG is 0.078 (95% CI 0.072–0.084), representing roughly a 17% reduction relative to the English score.

The gap is closely tied to language resources: net of Portuguese, the macro gap drops to 0.016, and language-level CLCG is negatively correlated with Joshi resource class (rho = -0.594, p = 0.015, n = 16). Blinded paired human evaluations confirmed the pattern, with higher-resource responses preferred in 61.6% of decisive judgments (estimated preference probability 0.655, 95% CI 0.558–0.741). The authors argue that English-centered benchmarks overstate model quality for users of low-resource languages. The paper, submitted to Computational Linguistics, spans 55 pages with 17 figures and is available on arXiv (2608.06506). For developers, the takeaway is direct: don't assume English benchmark performance will transfer to Spanish, Hindi, Swahili, or other languages. Multilingual deployment needs language-specific validation, targeted fine-tuning, and larger investments in low-resource evaluation data.