[P] Clip to Grok Update: Weight Norm Clipping now 39–249× | 6 Tasks (mod arithmetic, mixed ops, S5 permutation) | max_norm Measured Per Task

Independent researchers behind the 'Clip to Grok' project have published a significant update demonstrating that a simple weight norm clipping technique can dramatically accelerate neural network training on algebraic tasks. Their method applies per-row ℓ₂ clipping to decoder weights after every optimizer step, using no weight decay and requiring no extra memory. The team expanded their testing from modular multiplication to six distinct algebraic tasks: four modular arithmetic operations (addition, subtraction, multiplication, division mod 97), a mixed task combining all four operations, and S5 permutation composition—a non-abelian group with 120 elements.

Results show staggering speed improvements: median steps to 95% validation accuracy dropped from 35,040 to just 550 for modular multiplication (66x faster), and most dramatically, from 390,896 steps to 1,348 for S5 permutation composition—a 249x acceleration. The researchers discovered that optimal clipping norms (max_norm) correlate with algebraic complexity: inverse-dependent operations like division favor tighter norms (1.5–1.75), while direct operations like multiplication tolerate looser norms (up to 2.0). The non-abelian S5 structure required the tightest clipping at exactly 1.0, with performance degrading rapidly above 1.25.

The team conducted extensive experiments across 2,126 Adam runs, 7,137 Lion runs, and 2,125 SignSGD runs using unique seeds for statistical significance. They emphasize their findings are specific to algebraic tasks and may not transfer directly to other domains like natural language processing. The implementation is available on GitHub, and the researchers are seeking arXiv endorsement for their paper, which details how this simple regularization technique can force networks to learn more efficient representations of algebraic structures.