New tolerance calibration boosts bug detection by 9.3% in tensor kernels
Dipankar Sarkar's method tightens tolerances up to 2,184x, catching 229 more bugs.
A new preprint on arXiv proposes a data-driven approach to calibrating mixed-precision tolerances for tensor kernel correctness tests. Author Dipankar Sarkar mined accumulated cloud GPU runs from the 26-entry gpuemu corpus across two dtypes (float16 and float32), yielding 8,076 element-wise error distributions. Instead of using static, hand-picked absolute tolerances copied across tests, the method asks what tolerance the kernel's own correct implementation would justify. The answer is dramatically tighter: for example, the attention_triton fp16 kernel required a tolerance 2,184× stricter than the current default.
When tested on seven LLM-style buggy variants (paired with correct counterparts), the calibrated per-(op, dtype) tolerances improved bug-detection recall from 73.2% (1,805 of 2,467) to 82.4% (2,034 of 2,467) — an absolute gain of 9.3 percentage points representing 229 newly detected bugs. The false-positive rate rose minimally, from 0 to 20 out of 1,882 correct-control cases (+1.1 percentage points). The paper is 8 pages, LNCS format, and is part of a series (companion papers arXiv:2606.20128 and more to follow).
- Calibrated tolerances were up to 2,184× tighter than hand-picked defaults for attention_triton fp16.
- Bug-detection recall improved from 73.2% to 82.4% (+9.3 pp, 229 new detections) on seven LLM-style buggy variants.
- False positives increased by only 1.1 percentage points (0 → 20 out of 1,882 correct cases).
Why It Matters
Tighter tolerances catch more GPU kernel bugs in mixed-precision LLM workloads without flooding developers with false alarms.