Big AI Says It Speaks 100 Languages — Urdu Test Shows Cracks

A new study put three top AI chatbots — GPT-5.1, Qwen-3-Max and DeepSeek-3.1 — to a simple test: write short stories in Urdu, the language of more than 230 million people across Pakistan and India. The researchers collected 93 stories and read every one by hand, sorting the mistakes into nine categories covering grammar, meaning and culture. Researchers call Urdu a "low-resource language," meaning there is far less Urdu text on the internet for AI to learn from.

The results were rough. The models made basic grammar mistakes, mixed up word meanings, repeated the same phrases, and lost track of the plot. Most striking, the stories felt culturally empty — details about food, family, festivals and daily life were vague or borrowed from somewhere else. Think of a tourist writing a guidebook to your hometown after a two-hour visit: the words are roughly right, the feeling is not.

The researchers also tried to fix things by showing the models examples of good Urdu writing in the prompt — a technique called few-shot prompting (showing the AI what you want instead of just describing it). The cultural and context mistakes mostly stayed. So this is not a setting you can simply switch on. The study is small — one language, one task, 93 stories — but it echoes a wider pattern researchers keep finding.

Why should you care? If your job, business or family depends on AI helping in a language other than English — translation, customer service, homework help, medical or legal questions — you could get confident-sounding answers that are simply wrong, and you may never spot the errors. For companies serving non-English customers, that is a real business risk. And it is a reminder that when AI companies say "multilingual," they often mean English first and everything else later.