Nvidia's CUDA Gets Faster AI With New Fork/Join Trick
This could make AI training up to 30% faster, saving time and money.
PyTorch's CUDA work just got a new tool: torch.cuda.execute_on_streams, which adds fork/join execution with ordinary streams, green-context streams, or mixtures of the two.
According to the article, queued work is joined and caller state restored even when a callback raises. Same-device execution switches directly between streams and restores the caller once to reduce host-side overheads, while cross-device execution uses stream context managers to restore device and stream state.
The change was authored with an AI assistant and tested on Rubin with driver R610 plus CUDA 13.4 compat, where 56 tests passed. Pull request #199128 was approved, with ghstack dependency #199127.
- CUDA now supports fork/join execution, letting AI tasks run in parallel like multiple checkout lanes.
- This can speed up AI training and inference, potentially reducing costs for cloud AI services.
- It requires the latest Nvidia software and is mainly for developers, but benefits will reach end users.
Why It Matters
Faster AI means quicker responses, lower cloud costs, and new features sooner for everyone.