Prism Transformer's progressive head schedule boosts AI without extra cost

The standard Transformer allocates the same hidden dimension per head across all layers, a uniform design that the Prism Transformer identifies as a fundamental bottleneck. Early layers, with their restricted subspace, struggle to capture complex, high-dimensional contextual patterns. The Prism Transformer introduces a progressive head schedule: head count monotonically increases with depth. Shallow layers use fewer, exceptionally wide heads to model intricate local compositions, while deep layers employ many narrow heads to decompose those patterns into specialized linguistic features. Crucially, this rearrangement is parameter-neutral and compute-neutral – it uses identical weight matrices and FLOP budgets as the vanilla Transformer, adding zero training or inference overhead.

Tested at three scales (124M, 354M, 757M), the Prism Transformer consistently beats uniform baselines on validation loss and downstream zero-shot benchmarks including PIQA, HellaSwag, ARC-Easy, and WinoGrande. The paper shows that non-uniform subspace allocation unlocks latent capacity within the standard budget, enabling more effective use of model capacity. This is a simple architectural tweak that can be applied to any existing transformer model immediately, without retraining from scratch if the schedule is pre-defined. The work suggests that many current models are underutilizing their parameter budgets due to rigid head allocation.