Grok 4.20 leads AI bias: 20 LLMs favor some faiths over others

A new paper from arXiv (2605.22975) examines whether large language models treat religious conversion queries symmetrically. The answer is a clear no. Researchers tested 20 LLMs—including open-source and commercial models like Grok 4.20—across 182 religion pairings using a simulated user asking for advice on hypothetical faith transitions. They employed a human-verified LLM-as-a-judge framework to measure the language models' encouragement or discouragement. Results showed that models reliably favored conversions toward Catholicism, Baháʼí, and Sikhism, while discouraging conversions toward Atheism, Agnosticism, and Jehovah's Witnesses. Grok 4.20 exhibited the strongest asymmetries, but all models showed persistent, reproducible biases that remained stable across multiple question phrasings and dataset variations.

The authors note that the pattern of preferences differed per model provider, but asymmetry itself was a robust property—not an artifact of scoring. This raises significant concerns: AI systems deployed en masse can subtly influence personal faith decisions, particularly for users seeking guidance on life-changing conversions. The paper emphasizes that any imbalances reproduced at scale have real-world implications. With 29 pages and 16 figures, the study joins a growing body of work on AI value alignment and fairness, extending it into the sensitive domain of religious belief. The findings challenge the assumption that LLMs provide neutral advice and highlight the need for transparency in how models handle value-laden queries.