Research debunks AI concept dimension measurement claims

A team of researchers from Carnegie Mellon, MIT, and other institutions has published a groundbreaking paper that challenges fundamental assumptions about measuring concept dimensions in neural networks. The study, titled 'Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension,' demonstrates that common methods for counting concept directions (like iterative erasure) produce results that change under information-preserving transformations.

The research shows that while model-defined quantities like generating dimension and minimum guarding rank remain stable, procedure-defined metrics such as stopping count and cumulative edit rank vary dramatically with reparameterization. For example, an invertible shear transformation changed the cumulative Euclidean erasure count from 1 to 2 while preserving all predictions. The findings call into question decades of interpretability research that relied on these unstable metrics as semantic dimensions.

The team conducted experiments across multiple architectures including frozen V-JEPA2 features and found that even when rank-zero predictions remained unchanged, later Euclidean trajectories varied significantly under practical optimization. Their controlled experiments stress-test these measurement procedures rather than estimating true concept dimensions, suggesting that current interpretability techniques may be measuring procedure artifacts rather than intrinsic model properties.