New Study Reveals Persona Prompting Biases in LLM Scholar Recommendations

A new preprint titled "Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation" benchmarks how LLMs recommend academic experts when given different persona prompts. The team—Annabella Sánchez-Guzmán, Lukas Eberhard, Denis Helic, and Lisette Espín-Noboa—tested 43 LLMs by varying the prompt’s language, location, and role-task framing, as well as context like field, seniority, and recommended list size (k). They compared the LLM-generated scholars against Semantic Scholar’s ground truth across six scientific disciplines, measuring both technical quality (factuality, coverage) and social representativeness (diversity, parity).

Key results show that basic technical quality is primarily determined by model choice, while factuality and parity are more influenced by contextual elements like the user’s academic field or career stage. Crucially, diversity of recommendations is most affected by the location specified in the prompt: prompts set in South Africa produced less factual lists, whereas Japan-prompts yielded highly factual but homogeneous lists skewed toward highly productive scholars. The authors argue that persona prompt design is a nontrivial axis of LLM-based scholar discovery and must be systematically audited alongside model choice to ensure fair representation across global research communities.