[D] Quantified analysis of 2,218 Gary Marcus claims - two independent LLM pipelines, scored against evidence

A new quantified analysis provides a data-driven look at the prolific commentary of AI critic Gary Marcus. Researcher Dave Goldblatt built the 'Marcus Claims Dataset,' scoring every testable claim from Marcus's 474 Substack posts—totaling 2,218 assertions. Using two independent LLM analysis pipelines powered by Claude Opus 4.6 and ChatGPT Codex, followed by a reconciliation layer to compare outputs, the project found that among assessable claims, 52% were supported by evidence, 34% received mixed assessments, and only 6.4% were directly contradicted. The project, built in a single session and fully documented on GitHub, highlights how falsifiability drives the results, with nearly 20% of claims being inherently unprovable.

The detailed breakdown reveals a stark split in Marcus's track record. His specific, technical observations—such as those concerning LLM security vulnerabilities, the quality of OpenAI's Sora model, and the readiness of AI agents—scored remarkably high, with 88% to 100% support and zero contradictions. In contrast, his broader, more speculative predictions labeling the AI field a 'bubble' or 'scam' formed the single worst-performing cluster out of 54 categories analyzed. The methodology suggests a pattern where accurate, falsifiable technical calls resolve and disappear from discourse, while unfalsifiable claims accumulate. It's crucial to note the analysis is entirely LLM-scored and not human-verified, positioning it as a provocative tool for debate rather than a definitive verdict.