Attempting to Quantify Chinese Bias in Open-Source LLMs

Independent researcher Ethan Le Sage has developed a novel methodology to systematically measure political bias in Chinese large language models. Using Wikipedia's 7 million articles as source material, he generated 32,271 potentially controversial questions, then tested 250 high-risk questions across five LLMs: OpenAI's GPT-o1-20b, Mistral's Ministral-14b, Alibaba's Qwen3-32b, Z.ai's GLM-4.5, and Minimax's M2.7. The questions covered topics from religious freedom to capital punishment, designed to probe where Chinese models might exhibit censorship or bias.

Using GPT-o1-120b as an automated judge, the study found Chinese models scored significantly higher on bias metrics. Alibaba's Qwen3-32b showed the highest bias at 2.38 average score, followed by Z.ai's GLM-4.5 at 2.28—both approximately 3x higher than Western models. The bias manifested through answer refusal, fact reframing, and selective information presentation. Importantly, the research acknowledges its Western-centric perspective but highlights clear systematic differences in how Chinese versus Western LLMs handle sensitive topics.

The methodology employed multiple filtering stages including cheap LLM screening and expensive model validation to create a cost-effective benchmark. While limited by budget constraints and judge model bias, this approach provides the first quantitative framework for comparing political alignment across different LLM families. The findings suggest Chinese AI censorship extends beyond obvious topics like Tiananmen Square to broader human rights and governance discussions.