AI Doesn't Favor Male Artists, Study Finds — But the Test Is Too Blunt

AI models like CLIP are the quiet workhorses behind image search, photo tagging, and content filters. They learn by pairing pictures with captions scraped from the web, which means they can soak up human stereotypes along the way. To find out whether that happens in the art world, researchers ran two CLIP models against 1,500 objects from the Metropolitan Museum of Art's free online collection. They asked each model to score artworks on words like "masterpiece," "quality," and "influence," then checked whether the scores shifted based on whether the artist was a man or a woman.

At first glance, the answer was no. Scores for male and female artists landed almost on top of each other, and the differences were not statistically meaningful. Even after adjusting for things like the medium used, when the piece was made, and its shape, artist gender still had no measurable effect. That sounds like good news — and it partly is. It suggests these models aren't applying a crude "men make better art" shortcut when they look at museum images.

But the researchers are careful not to declare victory. Their own numbers show the test barely explained anything: the model's reasoning was so noisy that it behaved almost like a coin flip dressed up as judgment. A model can look unbiased simply because it isn't paying close enough attention to catch the difference. The authors also flag a second problem: 41.2% of the Met's holdings have no recorded artist, so any audit that skips them is measuring a filtered, surviving slice of history, not the whole picture. Bias could easily hide in the parts nobody can label.

The takeaway for anyone outside a research lab is that "we tested the AI and it was fair" is a much weaker claim than it sounds. Museums, hiring tools, dating apps, and stock photo sites all lean on this kind of image-and-text AI. The study argues that honest audits need to control for messy real-world factors, test for equivalence rather than just absence of a finding, and be upfront about whose data went missing.