Early-sign trick skips 86% of binary neural net compute

Binary neural networks are a key enabler for low-power, small-footprint inference on edge devices. In these networks, activations are binarized to +1/-1, turning dot products into simple sign-controlled additions and subtractions. But every neuron still accumulates all of its inputs—even though only the sign matters. As the sum grows, its final sign often becomes predictable long before the last term is added, meaning the remaining operations change the value but not the output.

To exploit this, researchers propose a post-training threshold method that characterizes running accumulation behavior on the training dataset, then stops computation early when the sign is highly predictable. No parameters are retrained, so it's a drop-in efficiency boost. On VGG11 applied to CIFAR-10, the approach removes 86.6% of accumulation terms in the deepest convolution for a 0.37-point accuracy drop, and 25% of full-network arithmetic when applied to the three deepest convolutions simultaneously for a 1.36-point drop. This offers a practical way to speed up binary neural network inference on constrained hardware.