Your AI Chatbot Catches Lies Worse in Chinese Than English
If you fact-check with AI in a second language, it may miss the believable lies.
Researchers built a benchmark called SWORD that tests whether AI models consistently reject factual errors across languages, generating well-formed but factually incorrect statements in eight widely spoken languages by deliberately distorting Wikidata triples. One finding stood out: models counterintuitively scored higher on semantically plausible distortions than on nonsensical random substitutions, suggesting they lean on distributional familiarity rather than genuine factual verification. Another: models with comparable baseline accuracy across languages degraded substantially on (East) Asian languages when given distorted statements, with cross-lingual performance gaps reaching up to 28 percentage points — a 49% relative reduction in some models. Multilingual factual reasoning, the authors argue, involves asymmetric capabilities that aggregate accuracy metrics systematically obscure.
- AI is better at spotting obvious nonsense than convincing-sounding wrong facts — so the subtle errors slip through.
- In Chinese, Japanese and Korean, error-catching dropped by up to 28 percentage points compared with other languages.
- Normal AI tests only check whether the model picks the right answer, which is why this gap stayed hidden until now.
Why It Matters
Using AI to fact-check in a non-English language? Don't fully trust it — it misses the believable errors.