SID-MLP Accelerates Generative Recommendations 8.74x with MLP Distillation

Generative recommendation models that use Semantic IDs (SIDs) have shown strong potential, but their practical deployment is bottlenecked by the high inference latency of beam-expanded autoregressive decoding. In a new paper submitted to arXiv, researchers Zitian Guo, Yupeng Hou, Clark Mingxuan Ju, Neil Shah, and Julian McAuley identify a key insight: the hierarchical structure of SIDs means prediction difficulty drops sharply after the first token, making repeated attention computations in standard Transformer decoders highly redundant. This observation drives their proposal of SID-MLP, a lightweight MLP-centric distillation framework that fundamentally simplifies the decoding paradigm. Instead of complex step-by-step attention, the model captures global user context in a single operation decoupled from sequential token prediction, then uses position-specific MLP heads distilled from a heavy autoregressive teacher. Extensive experiments show SID-MLP matches the accuracy of its teacher while accelerating inference by 8.74x, and the distillation strategy acts as a plug-and-play accelerator for various backbones and tokenizer settings.

Building on this, the team introduces SID-MLP++, an extension that replaces the Transformer encoder entirely to unlock further latency reductions—at the cost of a speed-accuracy trade-off for full encoder replacement. The paper highlights that decoder-side MLP distillation is an effective acceleration path for structured SID recommendation, while encoder replacement offers additional flexibility. For practitioners, this means generative recommenders—previously impractical due to high latency—can now be deployed efficiently in production settings. The work is particularly timely as recommendation systems increasingly move toward generative approaches, and the open-source contribution (code available on GitHub) allows teams to experiment with their own backbones. With inference speeds nearly 9x faster and no accuracy loss, SID-MLP could become a standard tool for accelerating generative recommendation models.