MatrixFSDP eliminates optimizer communication in ZeRO-3 for 54.6x speedup

Matrix optimizers such as Muon are known to improve convergence and token efficiency over standard coordinate-wise optimizers by orthogonalizing momentum-smoothed updates via Newton-Schulz. However, this requires access to the complete 2D matrix, creating a systems mismatch with ZeRO-3 sharding, which splits parameters across ranks to save memory. Existing approaches either reconstruct the full matrix at each optimizer step (paying weight-sized communication) or use ZeRO-1 owner placement with full parameters resident (exceeding memory limits).

MatrixFSDP proposes a third path: instead of changing the optimizer computation, it changes where ZeRO-3 shards live. For each 2D weight, one data-parallel rank owns the whole matrix while others hold empty shards; non-matrix tensors are packed into tail owners and remain on AdamW. The backward reduction naturally lands the full Muon input on the owner, allowing Newton-Schulz to run locally with no optimizer-step matrix communication. Forward and backward still materialize and reshard parameters using a custom runtime with MatrixShard metadata, balance-aware owner planning, deterministic owner-segment P2P collectives, owner-buffer pinning, and owner-shard checkpoint resharding. On 64 A100s, MatrixFSDP achieves 4.2x and 54.6x reductions in optimizer-step latency on one and eight nodes respectively, up to 2.15x end-to-end speedup, and supports model sizes that exceed 80 GB GPUs under ZeRO-1.