SET Framework Boosts GPU Performance Up to 44% with Smarter Scheduling
New CUDA runtime cuts synchronization overheads by 55% using event-chaining and work-stealing.
Deep Dive
Zhengxiong Li, Tsung-Wei Huang, and Umit Ogras propose SET, a CUDA runtime framework for task-parallel pipelines. It uses multi-stream event-chaining and work-stealing to minimize host-device synchronization delays and kernel scheduling overheads. Per-stream buffers ensure memory safety for concurrent jobs. Benchmarks show 1.15–1.44x speedup and 18–54% lower scheduling overhead versus state-of-the-art CUDA graphs. Accepted at Euro-Par 2026.
Key Points
- Achieves 1.15–1.44x speedup on real-world GPU workloads vs. state-of-the-art CUDA graphs.
- Reduces scheduling overhead by 18–54% using multi-stream event-chaining and work-stealing.
- Introduces per-stream buffers to ensure memory safety for concurrent in-flight jobs.
Why It Matters
Unlocks latent GPU performance for high-throughput pipelines, reducing latency and cost in AI and HPC.