RSA study: untrained nets beat backprop only at low resolution

Representational similarity analysis (RSA) is a popular method for testing whether convolutional networks learn brain-like representations, especially when comparing biologically plausible learning rules like feedback alignment or STDP against backpropagation. These plausible rules don't scale well, so studies typically train small networks on 32x32 CIFAR images, then compare their representations to brain responses modeled at much higher resolution. In a new preprint, Nils Leutenegger shows this resolution mismatch can flip conclusions.

The author found that the apparent parity between untrained networks and backprop-trained networks at V1 disappears when the evaluation resolution is increased. The gap widens monotonically from -0.001 at 32px to +0.044 at 224px, and this holds across human fMRI, macaque electrophysiology, training trajectories, and even for ImageNet-pretrained models like ResNet-50 and Swin-Tiny. Four candidate mechanisms—resolution matching, Gabor/pixel structure, normalization state, and brightness statistics—were tested and excluded with bit-identical weight controls. A fifth experiment revealed that capping image detail at training resolution while inflating pooling removes ~90% of the effect, implicating image detail rather than pooling. A striking baseline: a single luminance value per image achieves rho=0.075 against the V1 RDM, essentially matching the untrained network's 0.076, showing how low-level statistics can dominate these comparisons.