New Study: AI Tools That Judge News Bias Have Biases of Their Own

News organizations, researchers and social platforms increasingly use AI to do something humans used to do: read thousands of articles and decide how they're "framed" — that is, which facts get emphasized, which photos get chosen, and which side of a debate gets the sympathetic treatment. It's cheap, fast, and works at a scale no team of humans could match. But a new study asks an uncomfortable question: what if the measuring stick has its own opinions?

The researchers, Antonela Tommasel and Markus Schedl, built a test using real news coverage from 2025 and 2026. For each major event, they collected articles from left-leaning, centrist and right-leaning outlets covering the same headline — plus the images that ran alongside them. Then they put these articles in front of AI models that can read both text and pictures, asking the AI to assess the framing and viewpoint.

The results were messy in a revealing way. The AI's judgments changed depending on things that had nothing to do with the news itself: whether an image was included, whether extra information like the outlet's name was visible, and exactly how the question was worded. In other words, the tool's output partly reflects the tool, not just the story. The authors don't say the AI is useless — they say anyone using it needs to disclose these quirks and treat the results as one signal among many, not as a verdict.

Their contribution is a kind of inspection checklist: a standard way to test AI news-analysis tools and report what's actually influencing the answers before making big claims about media bias. It's a reminder that "the AI said so" is not evidence — especially when the whole point is deciding what counts as fair.