Implementation lottery: single AI runs misrepresent idea quality by 43%

Automated research systems increasingly rely on experimental scores to decide which AI ideas to keep, transfer, or pursue. But a single run tests only one implementation of an idea—a structural mismatch that creates what Ning et al. call the 'implementation lottery': crediting that realization-level score as evidence about the parent mechanism can flip conclusions depending on which plausible implementation was sampled. In their paper, the researchers quantify this effect across 312 assignments on 13 tabular tasks and two coding-agent setups, finding that implementation variance is more than five times same-artifact rerun variance, and in one setup exceeds ten times. Winner reversal (the idea that wins under one implementation differs from the winner under another) occurs in 25.6% and 43.6% of decisions, and persists even after outcome-blind review filtering.

To address this, the authors propose the 'Idea Reliability Audit' (IRA), which validates and freezes candidate cards, samples fresh-session implementations, uses outcome-blind fidelity labels, and reruns saved artifacts. The IRA reports idea ICC and leave-one-implementation-out winner reversal. An exploratory diagnostic on three materials-regression workflows with a deterministic evaluator also finds implementation variation dominating the variance decomposition. These findings draw a sharp line between idea reliability—how consistently an idea performs across implementations—and best-of-N artifact utility. For automated research to produce trustworthy conclusions, evidence must cover multiple implementations before a score guides idea-level branching, transfer, or research memory.