Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

The paper, led by Muhammad Dehan Al Kautsar and 15 co-authors, addresses a critical blind spot in LLM evaluation: cultural reasoning in dialectal Arabic. While most Arabic benchmarks use short MSA snippets, real conversations happen in dialects tied to local customs, idioms, and social norms. The team built ArabCulture-Dialogue — a manually curated dataset of dialogues from 13 countries (Egypt, Morocco, Saudi Arabia, etc.) across 54 subtopics like greetings, food, and traditions. They created three tasks: multiple-choice cultural reasoning (selecting the culturally appropriate response), machine translation between MSA and dialects, and dialect-steering generation (producing dialectal continuations).

Experiments on models like GPT-4, Llama 3, and AraBERT revealed a consistent >15% performance drop on dialectal versions versus MSA across all three tasks. Dialectal reasoning was particularly weak for less-represented variants (e.g., Sudanese, Tunisian). The findings underscore that current LLMs lack the nuanced cultural knowledge embedded in regional Arabic speech, limiting real-world deployment in social bots, translation tools, and educational apps. The dataset and benchmarks are publicly released to spur research into dialect-inclusive AI.