The Subjectivity of Monoculture

A new research paper titled 'The Subjectivity of Monoculture' by Nathanael Jo, Nikhil Garg, and Manish Raghavan challenges fundamental assumptions about AI model behavior. The study argues that claims about large language models (LLMs) exhibiting 'monoculture'—where outputs agree strikingly often—are inherently subjective. This subjectivity stems from two critical decisions researchers must make: first, specifying a baseline null model to define what 'independence' should look like, and second, selecting the specific population of models and evaluation items. The paper demonstrates that different choices lead to dramatically different conclusions about whether models actually agree too much.

The researchers validated their theoretical framework through experiments on two large-scale benchmarks, showing that inferences about monoculture change drastically depending on context. For example, using a null model that accounts for item difficulty produces completely different results than previous approaches that didn't. This work fundamentally reframes monoculture evaluation from being an absolute property of model behavior to a context-dependent inference problem. The findings have significant implications for how we measure AI diversity, assess model independence, and interpret claims about algorithmic convergence in the rapidly evolving LLM landscape.