Research Log: Monet/PEER sparse experts

A deep dive into the Monet and PEER sparse expert architectures reveals several promising paths toward more efficient and interpretable AI models. The research demonstrates that PEER models can be losslessly distilled to int8 precision and further compressed to int4 with only minor performance degradation. Crucially, the int4 models can be trained using a novel technique involving a second int4 tensor as a gradient accumulation buffer with stochastic rounding, allowing these values to be packed into int8 tensors. This approach enables training significantly larger PEER models on limited GPU VRAM, as the minor loss from int8 to int4 compression is offset by the substantial increase in experts that can fit per gigabyte of memory.

Beyond quantization, the research explores converting the better-training Monet models into the more interpretable PEER architecture through distillation, albeit with some computational overhead. For enhanced interpretability, each PEER expert can be distilled into a mixture of logical statements and mathematical functions using techniques from KAN 2.0 and Differentiable Logic Gates, which could also enable efficient CPU-based inference. To address the 'attention sink' phenomenon—a weakness for both architectures—a practical workaround pairs sparse expert layers with a small, shared feedforward MLP (dimension 128) with tied weights across layers, preserving most interpretability while solving the performance issue. The investigation also touches on advanced concepts like 'Metarouters' (MoE where each expert is itself a pool of sparse experts) and the use of JumpReLU gates for adaptive expert activation, pointing toward more specialized and efficient large-scale models in the future.