GSM-Plus-BN benchmark exposes LLM gaps in Bengali math reasoning

A team of researchers (Bidyarthi Paul et al.) has released GSM-Plus-BN, the first perturbed benchmark for evaluating mathematical reasoning in Bengali (Bangla) across large language models. Derived from the English GSM-Plus dataset, each of 1,000 seed questions was translated by human translators and augmented with 8 perturbation variants, creating a total of 9,000 evaluation samples. The benchmark tests true comprehension beyond pattern matching by systematically altering problem elements like numbers, variables, and wording.

Six open-source models were assessed: Qwen3-32B, Llama-3.1-8B, Llama-3.3-70B, Llama-4-Scout-17B, GPT-OSS-120B, and GPT-OSS-20B. Under standard prompting, GPT-OSS-20B led with 96.08% accuracy on seed questions, but larger models (Llama-3.3-70B, GPT-OSS-120B) proved more robust to perturbations. Chain-of-Thought prompting boosted performance for most models, yet a substantial accuracy gap remained compared to English benchmarks, underscoring the inherent difficulty of Bengali mathematical reasoning for current LLMs. The dataset and baselines provide a critical resource for developing equitable AI for over 230 million Bengali speakers.