[P] Yet another garage model - Prisma: Interpretability-Inspired Architecture

Independent AI researcher y3i12 has open-sourced Prisma, a novel transformer architecture that diverges from mainstream designs like GPT and Llama. The model introduces several interpretability-inspired modifications, including attention and output weight sharing to reduce parameters, Word-Relative Rotary Position Embedding for improved positional understanding, and a novel nested gating mechanism in the feed-forward network. This last innovation transforms the standard FFN operation W2 @ (W1 @ x * silu(W3 @ x)) into W2 @ (W1 @ x * silu(W3 @ x * silu(W4 @ x))), adding computational depth without dramatically increasing parameter count.

Trained on just 30 billion tokens from OpenWebText and FineWeb-Edu using a single NVIDIA H100 GPU, Prisma demonstrates remarkable efficiency gains. The architecture achieves approximately 25% better data efficiency compared to standard transformer designs while maintaining competitive performance on reasoning benchmarks including ARC-E, ARC-C, PIQA, BoolQ, and HellaSwag. This 'garage model' approach proves that architectural innovation—not just scaling—can yield significant improvements, potentially offering more efficient paths forward for the field beyond the current compute-intensive paradigm.