α-FLOPs formula fails on new GPUs: replication study reveals hardware quirks

In a new arXiv paper, Enrique Barba Roque and Luís Cruz set out to replicate a previous study that introduced α-FLOPs, a formula for estimating AI model execution time from floating-point operations. The original study argued that raw FLOPs are misleading because some layers parallelize better than others. The replication team tested this on newer, more powerful hardware and found the core thesis still holds: spatial dimensions (like image width and height) are easier to parallelize than kernel dimensions in convolutions. However, their fine-grained measurements revealed something the original study missed—modern hardware exhibits substantial instabilities in execution time, including sudden jumps and oscillations, which the α-FLOPs formula generally underestimates.

This negative result doesn't invalidate the original study's empirical findings, but it does show that α-FLOPs isn't reliable as a universal predictor across hardware generations. The replication also exposed gaps in the original materials: missing dependency details and unclear regression data. The authors stress that hardware-dependent AI efficiency research needs complete, reproducible packages. To that end, they've released their full replication package, hoping to encourage more rigorous comparisons and real-world performance assessments.