Recommender bias traced to long-tailed item margins, not Transformers

A new paper from Yang Cheng challenges the common assumption that representation anisotropy in recommender systems is primarily caused by Transformer architectures. Instead, the author identifies a more general source in the conditional training distribution of dot-product softmax decoders. The research proves that for any such encoder, the population-optimal score decomposes into three components: pointwise mutual information, an item-marginal term log p(i), and a context-dependent offset. After centering, this item-marginal term produces a context-shared rank-one score component, while time-varying marginals induce a low-rank popularity subspace. This means the apparent embedding collapse is actually a decoder-level consequence of long-tailed item marginals, not a property unique to Transformers.

The experimental validation is striking. On synthetic data and public Alibaba and Tianchi interaction logs, separating log p(i) from the learned dot product reduced the measured popularity-aligned score energy by 98.6% in a matched intervention. Permutation tests confirmed that this reduction is specific to the empirical popularity direction and not due to random noise. Importantly, the author notes that this score-level result does not imply universal embedding collapse because its transfer to embeddings depends on factorization geometry. These findings provide a clean, theoretically grounded explanation for a class of apparent representation degeneration, offering a direct path to debiasing recommender systems by explicitly handling the popularity component.