UNC's Ledger Residuals prove massive activations are no artifact

Trained transformers reliably develop massive activations—a small set of hidden dimensions with magnitudes far above the median, typically concentrated on the sequence-start token. Researchers have debated whether these outliers are just an artifact of the residual stream being forced to serve both as a scratchpad for intermediate computation and as the final readout channel. Maruthi Vemula from UNC Chapel Hill directly tested this with a new architecture called Ledger Residuals. It splits the residual stream into two separate streams: a mutable scratch stream (Deliberation) that intermediate layers may freely overwrite, and a protected, decode-only accumulator (Commitment) that holds the representation the model reads out. The hypothesis was simple: if massive activations exist only because one stream is overloaded, then providing a dedicated answer channel should eliminate them.

At model scales of 160M and 290M parameters with matched loss, the result was clear: massive activations rebuilt inside the protected Commitment channel. The reconstructed outliers were smaller in magnitude than in a standard transformer but more sharply concentrated on the start token. Moreover, applying stronger sparsity penalties made them more persistent and more concentrated, rather than removing them. This indicates that massive activations are architecturally robust—they re-emerge in whichever representation the model decodes from, suggesting they serve a functional role rather than being incidental. The paper releases code and measurements, offering a concrete benchmark for future work on transformer internals and outlier dynamics.