Claude 3.5 Sonnet leads in detecting misleading YouTube thumbnails with 93.8% accuracy

A new research paper presents a novel multi-modal detection pipeline that uses Large Language Models (LLMs) to flag misleading YouTube thumbnails, a pervasive problem eroding user trust. The team constructed a comprehensive dataset of 2,843 videos from eight countries, including 1,359 misleading videos that collectively amassed over 7.6 billion views, providing a unique cross-cultural perspective. Their pipeline integrates video-to-text descriptions, thumbnail images, and subtitle transcripts to holistically analyze content. They evaluated four frontier LLMs—GPT-4o, GPT-4o Mini, Claude 3.5 Sonnet, and Gemini-1.5 Flash—as well as open-weight vision-language models LLaVA-v1.5 and Qwen2.5-VL-7B-Instruct to test generalizability.

Among all models, Claude 3.5 Sonnet consistently showed the strongest performance, achieving 93.8% accuracy, over 92% precision, and recall exceeding 94% in certain scenarios. The team also conducted a careful failure analysis to understand when LLMs struggle with misleading thumbnails. Their findings have direct implications for content moderation, user experience, and the ethical considerations of deploying such systems at scale. The paper, accepted to the 21st International AAAI Conference on Web and Social Media (ICWSM 2027), paves the way for more transparent, trustworthy video platforms and stronger content integrity for global audiences.