DMuon: New distributed optimizer matches AdamW latency with 3x speedup
Open-source DMuon reduces optimizer overhead by up to 163x, enabling efficient Muon training at scale.
Matrix-orthogonalization-based optimizers like Muon have shown strong convergence in modern deep learning, particularly for large-scale models. However, their costly Newton-Schulz iterations made distributed training impractical — vanilla Muon required more than 2x the cost of forward and backward passes. DMuon, developed by Vincent Chen, Starrick Liu, and colleagues, solves this by efficiently distributing Muon's coupled weight matrix updates across multiple devices while maintaining near-AdamW levels of per-step latency.
DMuon achieves a 1.48x to 3.01x speedup in end-to-end step time and a dramatic 6.85x to 163.00x speedup in optimizer step time compared to naive Muon implementations. The optimization works across both embodied foundation models and large language models (LLMs). As an open-source module that drops into existing pipelines without framework modifications, DMuon makes Muon's theoretically superior convergence practically accessible for distributed training, potentially accelerating training of large-scale AI systems while reducing computational costs.
- DMuon is an open-source distributed implementation of the Muon optimizer that integrates as a drop-in module without framework modifications.
- It achieves 1.48x–3.01x speedup in end-to-end step time and 6.85x–163.00x speedup in optimizer step time over vanilla Muon.
- Per-step latency is brought to near-AdamW levels, enabling efficient scaling for both embodied AI and large language model training.
Why It Matters
DMuon drastically cuts training time for large models while preserving Muon's convergence benefits — a practical breakthrough for distributed AI.