Human-LLM collaboration builds Spanish stereotype dataset for fairer AI

A team led by Weicheng Ma and John Guerrerio at Dartmouth has developed a cost-efficient human-LLM collaborative framework to create culturally specific stereotype datasets, demonstrated with EspanStereo — the first comprehensive Spanish-language stereotype benchmark covering over 20 Spanish-speaking countries across Europe and Latin America. The framework leverages LLMs to generate candidate stereotypes from existing literature and cultural sources, then relies on in-country annotators to validate each item, ensuring cultural relevance and accuracy. This hybrid approach dramatically reduces the high cost and time typically required for manual annotation while capturing nuanced biases that English-centric resources (e.g., StereoSet, CrowS-Pairs) miss.

When the researchers evaluated several popular Spanish-supporting LLMs (including GPT-4, Claude, and Llama) using EspanStereo, they found stark variations in stereotypical behavior across different countries — for example, certain models exhibited stronger gender stereotypes for Spain than for Mexico, and regional biases about socio-economic status varied widely. The work, published at EMNLP 2025, provides a scalable blueprint for building multilingual, culturally specific bias benchmarks, directly addressing the long-standing gap in stereotype research outside English-speaking contexts. The authors have released EspanStereo publicly to enable broader cross-cultural bias evaluation in NLP.