Causal discovery benchmark inflated: zero-parameter compressor beats tuned methods

Stienstra's 'One Ruler' paper directly addresses a persistent credibility gap in causal discovery benchmarks. The Tuebingen cause-effect pairs are a standard testbed, but every method has been evaluated under different protocols—varying pair subsets, weightings, model selection, and decision rates. By running every method himself on the identical 102 pairs with one strict rule (no tuning, forced decision for every pair), Stienstra creates a level playing field. As a clean reference, he introduces sorted-conditional compression: quantize, sort, first-difference the data, then feed it to bz2. This baseline has zero fitted parameters yet reaches 74.7% weighted accuracy (p = 3.7e-7), outperforming many far more elaborate causal inference methods.

Under the common ruler, the ranking shifts dramatically. RECI's faithful re-run lands at 70.7%—inside the original authors' error bar, not the 77.5% often quoted (traced to a mis-copied cell). SLOPE's published 82.4% is a decided-subset figure; forced-decision yields 77.2%, and the compressor's 76.0% on the same 100 pairs is within noise (McNemar p = 0.39). Methods cluster in the low-to-mid 70s, and the zero-parameter compressor ties the strongest of them. Stienstra also shows that compression score magnitude is a model-free confounding flag (p = 2.8e-68), and a pre-registered falsification test bounds the method's theoretical interpretation. The paper releases code, pre-registrations, and per-pair outputs for full reproducibility.