PyTorch 2.13 delivers 12x speedups on Mac and 4x memory cut for LLM training

PyTorch 2.13 is a major step toward a unified, production-ready framework. FlexAttention now runs natively on Apple Silicon (MPS) with up to 12x speedups over SDPA on sparse attention patterns, and gains a deterministic backward pass on CUDA for reproducible gradient computation—with less than 1% overhead at long sequences. The new CuTeDSL “Native DSL” backend gives Inductor a second high-performance code path alongside Triton, enabling CUTLASS-grade GEMM kernels with faster compilation. For large-vocabulary language models, the fused nn.LinearCrossEntropyLoss combines final prediction and loss computation to reduce peak GPU memory usage by up to 4x, a critical optimization for practitioners scaling LLM training.

On the distributed side, torchcomms—a new communications backend—improves fault tolerance, scalability, and debuggability for large-cluster training. FSDP2 now overlaps reduce-scatter and all-gather operations via a dedicated process group, increasing throughput. Platform support expands: ROCm gains AOTriton 0.12b with native HIP CMake, Arm adds Armv9-A torch.compile targeting, and Intel XPU exposes new device telemetry APIs. ExecuTorch is integrated into PyTorch Core, making on-device inference a first-class capability. With 3,328 commits from 526 contributors, PyTorch 2.13 drives performance across GPUs, CPUs, and edge hardware.