Study: ChatGPT, Claude, Gemini, Grok, DeepSeek all copy news source bias

A Reddit user conducted a controlled experiment to test whether major AI models produce neutral summaries of news articles or inherit the source's political framing. They fed six immigration-related articles from left-leaning, center, and right-leaning outlets to ChatGPT, Claude, Gemini, Grok, and DeepSeek using identical neutral prompts. The resulting 30 summaries were manually coded for neutrality, accuracy, completeness, emotional language, and framing. The key finding: every model consistently reflected the source article's political slant. Left-leaning articles produced more negative summaries; right-leaning articles produced more positive summaries; only center-source articles yielded clean, neutral results. The summaries sounded objective when read in isolation, but subtle choices in emphasis, omission, and tone systematically shaped reader understanding toward the source's bias.

Claude 3.5 emerged as the strongest performer overall, scoring highest on neutrality and factual accuracy. Grok showed notable strength in completeness, capturing more details from the original articles. ChatGPT occasionally cut corners, omitting context that affected balance. The study’s author emphasized important caveats: only six articles were tested, coding was done by a single person, and the topic was limited to immigration. They made the full dataset available (Excel workbook with all summaries, rubric, and notes) on GitHub for replication and critique. While not definitive proof of systemic bias, this exploratory work raises a critical question for professionals who increasingly rely on AI-generated news digests: are we getting objective synthesis or source framing laundered through a convincingly neutral voice?