QuantiBias reveals quantization silently amplifies LLM bias by 24-27%

Emilio Ferrara's new benchmark, QuantiBias, reveals a critical safety gap in quantized large language models. Almost every deployed LLM is quantized—compressed from full precision to reduce memory and latency—under the assumption that this step is harmless. Standard safety evaluations check refusal of harmful requests, avoidance of over-refusal, and unbiased multiple-choice answers. Quantized models pass all these checks. Yet when asked open-ended questions, the same models volunteer stereotypes in roughly one in four responses (~24-27% across compression levels), and this holds across all eight languages tested. The bias is a robust phenomenon that standard evaluations systematically miss.

Using a generative multilingual stereotype probe paired with refusal and multiple-choice controls, QuantiBias isolates open-ended generation bias. Tested on Qwen and Gemma backbones across five quantizer families and eight benchmarks, the study finds that quantizers allocate extra precision disproportionately to capability data that carries no bias-prevention signal. Notably, adding a reasoning step before answering roughly halves the bias effect on some quantizer families while doing nothing on others. The takeaway: every quantized model must be re-evaluated for open-ended bias, not just short-form safety checks.