MiniMax's MSA cuts LLM attention compute by 28x
New sparse attention method delivers 14x prefill speedup at 1M context...
MiniMax has unveiled MiniMax Sparse Attention (MSA), a breakthrough in handling ultra-long contexts for large language models. The core innovation is a blockwise sparse attention mechanism built on Grouped Query Attention (GQA), where an Index Branch scores key-value blocks and selects a top-k subset for each GQA group. This enables group-specific sparse retrieval while maintaining efficient block-level execution.
To translate this sparsity into real-world speedups, MiniMax co-designed MSA with a specialized GPU execution path that leverages exp-free top-k selection and KV-outer sparse attention. The result is dramatic: for a 109B-parameter multimodal model, MSA matches GQA performance while reducing per-token attention compute by 28.4x at 1M context. When paired with their custom kernel, MSA achieves 14.2x prefill and 7.6x decoding speedups on NVIDIA H800 GPUs. MiniMax has open-sourced the inference kernel and released a production-grade multimodal model using MSA.
- MSA reduces per-token attention compute by 28.4x at 1M context for a 109B-parameter model compared to GQA
- Achieves 14.2x prefill and 7.6x decoding speedups on H800 GPUs with co-designed kernel
- Open-sourced inference kernel and released production multimodal model (MiniMax-M3) on Hugging Face
Why It Matters
Enables trillion-token context windows on consumer GPUs while cutting costs by orders of magnitude.