MawForge runs MoE models locally by materializing experts on demand

Researchers have long struggled to run large sparse Mixture-of-Experts (MoE) language models on local hardware because the full model parameters, key-value cache, and runtime buffers often exceed available memory. Craig Opie's new paper, MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference, proposes a radical alternative: keep the full model on disk, hold only common tensors in RAM, and materialize specific expert tensors into a bounded execution cache only when they are routed to by the router network. This approach allows devices like MacBooks with unified memory to serve MoE models far larger than their physical RAM would normally permit.

MawForge's central finding is that it works well as a bounded execution mechanism and for measuring the trade-offs in local MoE inference, but it is not a cache-maximization policy—its effectiveness hinges on balancing expert reuse against resident footprint, as well as KV-cache size, quantization level, route locality, and macOS memory pressure. The system offers a practical path for running 100B+ parameter MoE models on consumer hardware, but operators must tune caching carefully to avoid thrashing. The paper is a significant contribution to on-device AI, potentially enabling powerful language models on laptops and edge devices without reliance on cloud servers.