Free AI Models Just Got Up to 12x Faster on Your Laptop
This could make AI chatbots and image tools run dramatically faster on everyday computers.
The ggml CPU backend now defaults to F32 src1 for F16 matrix multiplication: vec_dot_type(F16) is F32 and the dot product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32 src1), so src1 is no longer quantized to F16 and worst-case inputs that only fit in the F32 FP range are handled correctly. conv_2d/conv_3d now store im2col patches — the inner mul_mat src1 — as F32 and pass separate src0/src1 types to the inner mul_mat, because the previous F16 patches made the inner mul_mat widen src1 into the conv work buffer, corrupting the patches and crashing on large inputs.
conv_transpose_1d/2d (F16 kernel) keep src1 in F32 in the work buffer and use ggml_vec_dot_f16_f32, with work buffer planning updated accordingly. supports_op restores MUL_MAT support for F16 src1, which is now widened to F32, and ggml_vec_dot_f16_f32 gains GGML_SIMD support by converting chunks of x to F32 and reusing the F32 SIMD dot. The 256-bit fused path previously required AVX2 and FMA, so CPUs with AVX and F16C only fell back to chunked conversion and
- Free AI software ggml now runs up to 12 times faster on some computers by using a more accurate number format.
- This fixes crashes and errors that could happen with large images or complex calculations.
- You'll see faster responses from AI apps on your laptop, even without a high-end graphics card.
Why It Matters
Faster, more reliable AI on everyday devices means you spend less time waiting and more time getting things done.