[D] We tested the same INT8 model on 5 Snapdragon chipsets. Accuracy ranged from 93% to 71%. Same weights, same ONNX file.
The same ONNX model's accuracy varied wildly, from 91.8% on a flagship chip to 71.2% on a budget one.
Deep Dive
A user's on-device testing revealed a major AI deployment issue. The same INT8-quantized model, deployed via ONNX to five different Qualcomm Snapdragon chipsets, produced drastically different accuracy scores. Performance ranged from 91.8% on a Snapdragon 8 Gen 3 down to 71.2% on a Snapdragon 4 Gen 2. This 22-point gap, caused by NPU precision and runtime optimizations, is invisible in cloud benchmarks and poses a serious risk for developers shipping mobile AI.
Why It Matters
Developers can't trust cloud benchmarks; real-world mobile AI performance varies wildly by hardware, risking product quality.