Sharma & Vu expose looped LM 'readout blind spot' causing exploding norms

Looped language models (LLMs) that repeatedly feed hidden states back into computation have a dangerous blind spot: dense per-loop cross-entropy loss only supervises variables directly exposed by the readout mechanism, not the full recurrent state. Rituraj Sharma and Tu Vu demonstrate this 'readout blind spot' by showing that scale-invariant normalizations (RMSNorm, LayerNorm) hide radial scale from the immediate loss, while the residual recurrence continues to carry and update that same scale. In 44M and 128M parameter looped transformers without inter-loop normalization, using standard RMSNorm readouts allowed final hidden-state norms to balloon into the thousands or tens of thousands, despite achieving low perplexity at early exits.

The fix is twofold: either make scale visible to the loss (via scale-visible readouts or explicit norm penalties) or remove scale from the recurrence entirely (e.g., use scale-removing normalization between loops). With these architectural corrections, hidden-state norms stay in the tens, and perplexity improves at matched inference-depth operating points in variable-depth benchmarks. The result is a simple design rule: dense supervision trains exits; recurrent scale control requires either making scale visible to a loss or removing it from the loop. This is essential for stable deployment of looped models that trade compute for accuracy at inference time.