BiasIG: Benchmarking Multi-dimensional Social Biases in Text-to-Image Models

A research team led by Hanjun Luo has introduced BiasIG, a comprehensive new benchmark designed to systematically measure the multi-dimensional social biases embedded in text-to-image (T2I) generative models like Stable Diffusion, DALL-E, and Midjourney. Unlike previous benchmarks that often conflated different types of bias or focused narrowly on occupational stereotypes, BiasIG is grounded in sociological and machine ethics frameworks. It disentangles bias across four distinct dimensions, enabling a fine-grained diagnosis of how AI models perpetuate societal stereotypes. The benchmark uses a curated dataset of 47,040 prompts and features a fully automated evaluation pipeline powered by a fine-tuned multi-modal large language model, achieving accuracy comparable to human experts.

The researchers conducted extensive experiments on eight popular T2I models and three common debiasing methods, uncovering critical and often counterintuitive insights. Their findings reveal that interventions targeting specific protected attributes—like race or gender—frequently trigger unintended confounding effects on unrelated demographics, potentially creating new forms of bias. More alarmingly, the study found that current debiasing techniques exhibit a persistent tendency to shift model behavior from mere ignorance to active discrimination. The work advocates for a more precise, taxonomy-driven approach to fairness in AI-generated content (AIGC) and provides a theoretical framework for using BiasIG's metrics as feedback signals in future closed-loop mitigation systems. The benchmark is openly available, offering developers a robust diagnostic tool to build more equitable generative AI.