New Study Reveals Pipeline Choices Skew AI Interpretability Scores

A new preprint from van der Ben and colleagues systematically deconstructs the evaluation pipeline for sparse autoencoder (SAE) interpretability. The core finding: when researchers compare SAE architectures using autointerpretability scores (where an LM explains features and another LM scores the explanation), the scores reflect pipeline choices—not architectural merit. Across 4 metrics (simulation, detection, fuzzing, purity), 2 models (Pythia-160M, Apertus-8B), and 4 axes of methodological variation, methodological variance consistently exceeded architectural variance. Detection was the most stable metric; fuzzing was unreliable across all conditions. Worse, top-k feature rankings appear stable on average but individual features shuffle dramatically across corpus and draw conditions—a hidden instability that explanation similarity monitoring cannot catch.

These results have immediate consequences for the ongoing debate on SAE utility: cross-paper conclusions may simply compare evaluation pipelines, not the autoencoders themselves. At a time when reliable interpretability tools are critical for AI safety, unreliable evaluation slows progress. The authors contribute a variance decomposition approach, a Stability Check, and a Minimum Reporting Checklist to help future researchers account for these confounds. The paper suggests that the field needs to standardize evaluation protocols before claiming architectural improvements matter.