WAV residual routing boosts 48-layer Transformers with directional details

Standard residual connections in deep Transformers use fixed unit weights or content-dependent routing. Block Attention Residuals improved efficiency by routing over block-level summaries, but these summaries only capture low-frequency total displacement, missing directional structure like attention-vs-MLP imbalance and early-vs-late block dynamics. WAV v1, introduced by Kehan Wang, augments each block with two additional directional detail bases: a phase basis contrasting attention and MLP updates, and a split basis contrasting early and late sublayer updates. These are routed alongside standard block summaries through a depth-wise softmax mixer, with negative detail-source initialization and detached RMS matching to stabilize training.

Experiments on character-level TinyStories and Text8 show that WAV v1 provides depth-dependent benefits. At 12 layers, improvements are not consistent, but at 24 layers it becomes competitive, and at 48 layers it outperforms all baselines. Specifically, WAV v1 reduces validation loss from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8 relative to Block AttnRes, with negligible additional parameters. These results indicate that directional residual details are critical for scaling residual routing in deeper Transformers, suggesting a path toward more efficient training of very large models.