LLM Math Parameters: Partial Cross-Lingual Overlap, English Dominates

Researchers from multiple institutions (Shomali et al.) conducted a cross-lingual mechanistic analysis of mathematical reasoning in LLMs, accepted at the ACL Student Research Workshop 2026. They asked whether the parameters driving math performance are shared across languages or language-specific. By localizing and comparing model parameters for math tasks in different languages, they discovered that math-associated parameters exhibit partial overlap across languages. This overlap is strongest in the intermediate layers of the transformer architecture, suggesting these layers serve as a common hub for abstract mathematical reasoning. The study systematically extracted parameter sets for each language, revealing that English consistently produces the largest and most robust set of math-relevant parameters, likely due to its dominance in training data.

The findings indicate that LLMs' mathematical reasoning is neither fully language-invariant (i.e., using identical parameters for all languages) nor fully language-specific (completely separate parameter sets). Instead, there is a nuanced middle ground: a shared core of parameters (especially mid-layer) that supports math across languages, augmented by language-dependent parameter sets. Lower-resource languages yield smaller relevant parameter sets, which may explain performance disparities. The authors provide code and datasets to facilitate further research. This work has direct implications for improving multilingual LLMs—suggesting that targeted fine-tuning of intermediate layers could boost math performance in underrepresented languages without compromising English capabilities.