Why 7 Major LLMs Failed at Cross-Lingual Transfer — And What It Reveals About AI’s Real Weakness

A new study from Ahmed Haj Ahmed, Ruochen Zhang, and Alvin Grissom II challenges assumptions about cross-lingual transfer in large language models. The team fine-tuned seven LLMs ranging from 4B to 671B parameters (dense and Mixture-of-Experts architectures) on Arabic, then evaluated zero-shot reading comprehension on Semitic languages (e.g., Hebrew, Amharic) and non-Semitic controls (e.g., English, Turkish). Surprisingly, they found no evidence of Semitic-specific transfer: models with weak initial baselines improved dramatically on all languages tested, while strong-baseline models showed only marginal gains regardless of language family. This suggests the fine-tuning effect is not about learning linguistic relatedness but about aligning to the task format.

Further supporting this, a chain-of-thought (CoT) ablation showed that the same models benefiting most from fine-tuning also benefited equally from inference-time reasoning (CoT). Both mechanisms address task-format alignment rather than cross-lingual knowledge transfer. The paper, posted on arXiv (cs.CL/2606.19346), implies that current LLM fine-tuning for cross-lingual tasks may be overestimated in its linguistic transfer capabilities. Practitioners should focus on improving task alignment and reasoning rather than expecting automatic linguistic generalization from closely related languages.