Fixed Depth Supervision Beats Learned Halting Gates in Looped Transformers

Looped transformers boost test-time compute by reusing a recurrent block, but their adaptive depth mechanisms have been poorly understood. A new paper from Cambridge researchers dissects the problem, showing that learned halting gates entangle two critical functions: deciding when to exit and weighting intermediate states during training. This entanglement often corrupts the trajectory—the sequence of hidden states the model traverses—leading to suboptimal adaptive-compute performance.

The team tested their diagnostic framework on modular arithmetic, binary parity, and large-scale Ouro-1.4B and 2.6B checkpoints. Their key finding: fixed-prior depth supervision—which shapes the trajectory without an input-dependent halting policy—produces difficulty-aware intermediate states that naturally expose useful stopping signals. Simple post-hoc confidence readouts often match or outperform learned linear and MLP gates, and fitting gates on frozen trajectories reveals that the failure stems primarily from the trajectory induced by joint gate training, not from limited gate expressivity.

On the Ouro evaluations, pretrained ponder gates were competitive but not uniformly Pareto-optimal, and measured latency confirmed that reduced average exit depth translates into real inference-time savings. The paper reframes adaptive depth as a joint problem of trajectory formation and exit readout, rather than gate learning alone—a distinction that prior work has often left implicit. This insight could reshape how we design adaptive-compute transformers, emphasizing trajectory quality over gating complexity.