Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

A new research paper by Hailing Cheng, titled 'Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems,' tackles a core inefficiency in modern AI-powered recommendation engines. Current systems, known as Generative Recommenders (GR), model user behavior by interleaving item tokens (like a movie title) and action tokens (like a 'click') into a single sequence for a Transformer model. This approach doubles sequence length, creates quadratic computational overhead, and forces the model to disentangle unrelated signals, introducing noise.

The work proposes a principled reformulation that aligns sequence modeling with the underlying causal structure—specifically, that an item causes an action. It introduces two novel architectures, AttnLFA and AttnMVP, which eliminate interleaved dependencies. Instead of mixing tokens, these models explicitly encode the item-to-action causal link, reducing overall sequence complexity by 50%. This leads to cleaner attention mechanisms and more efficient computation.

Evaluated on large-scale product recommendation data from a major social network, the new models delivered significant gains. AttnLFA and AttnMVP outperformed standard interleaved baselines, improving evaluation loss by 0.29% and 0.80%, respectively, and showing gains in ranking metrics like Normalized Entropy. Crucially, these performance improvements came with substantial efficiency wins: training time was reduced by 23% for AttnLFA and 12% for AttnMVP. The findings suggest that explicitly modeling causality is a superior design paradigm for building scalable and efficient generative ranking systems.