Beyond Seeing Is Believing: On Crowdsourced Detection of Audiovisual Deepfakes

Soprano et al. (2026) investigated whether crowdsourced workers can reliably detect audiovisual deepfakes—increasingly realistic AI-generated manipulations that alter audio, video, or both. Using two benchmark datasets (AV-Deepfake1M and Trusted Media Challenge), they sampled 48 videos per set (96 total) and collected 10 judgments per video via Prolific, totaling 960 assessments. The study measured three tasks: authenticity classification, manipulation type identification (audio-only, video-only, or both), and manipulation timestamp localization.

Results show that crowd workers rarely flag authentic videos as manipulated (low false positives), but they miss a substantial number of actual manipulations—even when told to watch carefully. Inter-rater agreement was low across videos, meaning different workers perceive fakes differently. While aggregating judgments across multiple workers improved the reliability of the overall authenticity signal, it could not recover manipulations that the majority of workers consistently overlooked. This suggests that simple majority voting is insufficient for catching sophisticated deepfakes.

Modality attribution proved even more challenging. Workers who correctly identified a video as manipulated often misidentified which modality was faked. Joint audio-video manipulations were the hardest to recognize, with accuracy barely above chance. Timestamp reports for manipulations were also inconsistent, making it difficult to pinpoint exactly when a fake occurred. The authors conclude that while crowdsourced detection offers a scalable screening layer for online platforms, reliable modality attribution and detection of subtle fakes remain unsolved problems.

Overall, the findings have direct implications for misinformation moderation pipelines. Crowdsourcing can serve as a first-pass filter to flag potential fakes for expert review, but it cannot be relied upon as a standalone solution—especially against high-quality joint audio-video deepfakes. The researchers recommend combining crowd signals with automated detection tools and improving worker training to address the identified weaknesses.