Stimulus symmetries can distort neural code comparisons using RSMs

A new arXiv preprint by Farhad Pashakhanloo and Jacob Zavatone-Veth exposes a critical flaw in representational similarity analysis (RSA), a widely used method for comparing neural codes across networks or brain regions. RSA works by computing representational similarity matrices (RSMs) that summarize the geometry of neural activity patterns. The authors prove that symmetries in input stimuli can make different neural representations functionally equivalent yet produce distinct RSMs. This means two models solving the same task could appear to have different neural geometries solely due to input symmetries, not actual coding differences.

The team demonstrates this phenomenon in artificial neural networks trained on image data, where latent symmetries (e.g., translation, rotation) cause RSMs to drift during stochastic gradient descent. They also show that energetic regularization can produce sparse, drifting codes with the same issue. The paper's findings imply that RSA-based comparisons may be unreliable unless input symmetries are carefully controlled. For AI practitioners, this is a cautionary tale: popular benchmarking tools may overstate model dissimilarities, and researchers should consider symmetry-aware alternatives when evaluating representational alignment.