Meta's New AI Tool Makes Training Big Models 18% Faster
This could cut AI training costs and speed up your favorite apps.
AutoParallel is a PyTorch-native planner for distributed model training. It searches parallelization choices jointly across the training computation, expresses decisions using native PyTorch abstractions, and produces ordinary PyTorch programs without a separate execution runtime.
It can also compose automatic decisions with model-specific distributed code. Across dense, hybrid, and mixture-of-experts models on meshes of up to four dimensions and 128 NVIDIA H100 GPUs, AutoParallel matches or exceeds expert-written TorchTitan recipes on most configurations, with throughput gains of up to 18.3%, and discovers plans that uniform expert recipes do not express.
- Meta's AutoParallel automatically finds the best way to split AI training across many computers, beating human experts by up to 18% in speed.
- Faster training means AI companies can save money and time, potentially speeding up improvements to AI products you use.
- The tool is open-source and works with PyTorch, but still requires technical expertise to implement.
Why It Matters
Faster AI training could lead to quicker, cheaper AI improvements in everyday apps and services.