[R] LOLAMEME: A Mechanistic Framework Comparing GPT-2, Hyena, and Hybrid Architectures on Logic+Memory Tasks
Hybrid THEX model scores 0.738 vs GPT-2's 0.249 on multi-language tasks, showing attention and convolution have complementary strengths.
Stanford researchers developed LOLAMEME, a synthetic evaluation framework that systematically compares Transformer (GPT-2), convolution-based (Hyena), and hybrid architectures on tasks requiring logic, memory, and language understanding. Unlike previous mechanistic interpretability work using toy tasks, LOLAMEME tests real-world complexities like variable naming conventions, persistent memory, latent type systems, and mixed-language syntax. The team created two configurable programming languages (LoLa and MeMe) with different syntax and built THEX, a hybrid architecture that strategically replaces Hyena layers with GPT-2 attention blocks. Key results show THEX-13 achieving 0.738 accuracy on multi-language tasks versus Hyena's 0.492 and GPT-2's 0.249, demonstrating that attention and convolution have complementary strengths. These findings have direct implications for designing next-generation models like Mamba and StripedHyena.
- THEX-12 achieved 0.36 exact match accuracy vs. Hyena's 0.14 and GPT-2's 0.007 on tasks with global variables
- Hyena models memorize better than GPT-2 at moderate scale but collapse at 1000 variables
- Optimal attention layer placement in hybrid architectures varies significantly by task complexity
Why It Matters
Provides crucial design insights for next-gen AI models like Mamba and StripedHyena, showing how to combine attention and convolution effectively.