DeRes boosts CTR models 2x compute savings with dual-path residuals

Transformer-based Click-Through Rate (CTR) models suffer from a fundamental bottleneck at the residual connection: early user-interest signals get diluted layer by layer, the identity skip can't forget stale interests, and each layer only sees its immediate predecessor. To address this, a research team from a major social-media platform proposes DeRes, a novel architecture that decouples residual stability and adaptivity. Drawing inspiration from Dual Path Networks (DPN) and HORNN view of residuals, DeRes routes each layer through two parallel paths: an Identity residual path that preserves first-order feature reuse and gradient flow, and a Block Attention Residual path that attends over compressed outputs of all earlier blocks for high-order recall. A vector-wise gate determines per hidden dimension the weight given to each path. Additionally, the paper introduces Pointwise AttnRes, which replaces Softmax in cross-layer attention with SiLU, allowing multiple past blocks to be activated simultaneously and irrelevant ones to receive negative (forgetting) weights—better aligned with CTR's parallel multi-interest patterns.

DeRes was evaluated on three large-scale datasets: a proprietary industrial dataset with 331 million interactions from the social-media platform, plus Criteo (45M) and Avazu (40M). It outperformed twelve strong baselines, including OneTrans, TokenMixer-Large, UniMixer, mHC, and AttnRes, achieving up to +0.32% AUC improvement at under 5% extra FLOPs. More importantly, DeRes exhibits a markedly steeper compute-AUC scaling law (gamma=0.118 vs. 0.071 for OneTrans—a 1.66x gap), meaning an 8-layer DeRes matches the performance of a 16-layer OneTrans, roughly 2x compute savings at equivalent AUC. Ablation studies confirm that the dual-path design outperforms either single path alone, that the Identity path beats learnable residuals, and that SiLU activation outperforms Softmax. This work paves the way for more scalable CTR prediction without sacrificing accuracy.