PyTorch's register-resident topk kernel speeds up small-K operations by 4.45x

PyTorch's latest contribution to its native CUDA DSL stack introduces a register-resident topk kernel optimized for small K (16 and 32) and small N (up to 2048). Authored by Claude and merged into the main branch, this kernel fundamentally rethinks the data path: instead of staging keys through shared memory or global memory, it keeps all keys in registers and writes only the final K values to global memory. This design eliminates the latency and memory bandwidth overhead of intermediate storage, making it particularly effective when the radix-based kernel cannot run (K<64) and the one-CTA-per-row approach suffers from launch overhead.

The performance gains are substantial on NVIDIA B200 hardware. For K=16 across a range of N values (64–2048), the kernel delivers a geometric mean speedup of 2.24× with individual benchmarks ranging from 1.16× to an impressive 4.45×. For K=32 with N=256, the geometric mean speedup is 1.04× with no regressions across the entire eligible (K, N, M) grid. Beyond K=32, the kernel gracefully falls back to the existing radix kernel (for K=64–1024) or the aten implementation. The dispatcher in cutedsl_impl.py handles this automatically. A key design choice is losslessness: unlike some aggressive optimizations that sacrifice accuracy, this kernel preserves the full original index for each key, ensuring bit-exact output with deterministic tie order, at the cost of slightly lower raw throughput compared to quack's small-K topk.