Static PTX metrics catch structural GPU kernel bugs but miss semantic ones

A new paper by Dipankar Sarkar tackles the challenge of detecting regressions in GPU kernels by examining the relationship between static PTX metrics and measured runtime. The study pairs each kernel's static PTX characteristics — register count, spills, and instruction count — with CUDA-event-timed runtime across five distinct GPU classes: RTX 3060, A10, L40S, A100 SXM4, and H100 NVL. The key finding is that static and measured signals separate cleanly: for any given (correct, buggy) kernel pair, the per-pair delta in registers and instructions is identical across all five GPUs, while the measured delta in performance (%) varies significantly by hardware. This means static metrics offer a portable, hardware-independent signal.

The paper distinguishes two types of bugs. Structural bugs, like the gelu_triton_buggy variant that drops a leading 0.5 factor, change the kernel's work and produce clear deltas in PTX (8 fewer instructions, 8 fewer registers). The corresponding runtime delta on RTX 3060 is only +3.2%, well within run-to-run noise at sub-millisecond scale. Semantic bugs, such as softmax_triton_buggy which substitutes other=0.0 for -inf on the masked load, compile to byte-identical PTX — they are completely invisible to static metrics. The paper's bounded claim is that a static-PTX delta gate can serve as a portable pre-filter separating structural from semantic changes, while measured runtime deltas at this scale are hardware- and noise-sensitive and cannot substitute for static analysis.