Adding Typos Made Haiku's Accuracy Go Up

A viral AI experiment claimed that adding typos to prompts caused Anthropic's Claude 3.5 Haiku model to perform better, showing a surprising 22% accuracy increase on the BigCodeBench coding benchmark as typo rates rose to 16%. This contrasted sharply with larger models like Claude 3 Opus, whose accuracy dropped by 9%. The initial hypothesis was that harder-to-read text forced the model to 'try harder,' similar to the 'desirable difficulty' effect observed in human learning.

However, the researchers discovered the anomaly was specific to Haiku models and the BigCodeBench benchmark. Upon deeper investigation, they found the 'typo uplift' was an artifact of their custom grading harness. The harness was designed to extract only the last code block from a model's response for evaluation. As typo rates increased, Haiku's behavior shifted: it began producing single code-block responses about 20% more often, which happened to align perfectly with the grader's extraction method. When the researchers modified the harness to evaluate all generated code blocks, the accuracy boost completely disappeared, revealing no real performance improvement.

This case highlights the critical, often overlooked role of evaluation infrastructure in AI benchmarking. The researchers demonstrated that by simply tuning their grading harness—without changing the model or test questions—they could artificially 'improve' Haiku's score from 31% to 53%. This serves as a stark reminder that reported benchmark scores are lower bounds of capability, heavily influenced by design choices in the testing pipeline. The incident underscores the need for robust, transparent evaluation methodologies as the field pushes for more reliable model comparisons.