SORT: A Systematically Optimized Ranking Transformer for Industrial-scale Recommenders

A research team led by Chunqi Wang has introduced SORT (Systematically Optimized Ranking Transformer), a breakthrough model designed to bring Transformer architecture to industrial-scale recommendation systems. While Transformers have dominated large language models (LLMs), their application to ranking tasks has been limited by challenges like high feature sparsity and low label density in real-world data. SORT bridges this gap through systematic optimizations including request-centric sample organization, query pruning, and generative pre-training, enabling Transformers to effectively handle the complex, sparse data typical of platforms like e-commerce sites and content feeds.

The technical innovations extend to refinements in tokenization, multi-head attention (MHA), and feed-forward network (FFN) modules, which collectively stabilize training and increase model capacity. Crucially, the team optimized the training system to achieve 22% Model FLOPs Utilization (MFU), maximizing hardware efficiency. Extensive online A/B testing in large-scale e-commerce scenarios demonstrated remarkable results: SORT increased orders by 6.35%, buyers by 5.97%, and Gross Merchandise Value (GMV) by 5.47%, while simultaneously reducing latency by 44.67% and more than doubling throughput (+121.33%). This combination of improved business metrics and superior performance efficiency makes SORT a compelling solution for companies running recommendation systems at scale.