Open-Source AI Gets Speed Boost, Making Local Chatbots Faster

If you've ever used a free AI chatbot on your computer, you might have noticed it can be slow. That's often because the AI is running on your computer's main processor (CPU) instead of a powerful graphics card (GPU). A project called tinyBLAS, which is part of the popular open-source AI toolkit ggml, just got an update that makes these CPU-based AIs faster.

The update adds support for three number formats: BF16, FP16, and FP32. In simple terms, these are different ways to store numbers that AI models use. BF16 and FP16 are like using smaller boxes to store numbers, which saves space and speeds up calculations, while FP32 uses bigger boxes for more precision. Before, tinyBLAS could only handle certain parts of these formats efficiently; now it can handle the 'tails'—the leftover bits—much faster by using special CPU instructions called vectorization.

Why does this matter? Because it means AI models can run more efficiently on the computer you already own. That could make local AI assistants more responsive, reduce the need for expensive cloud servers, and lower costs for companies that provide AI services. It also helps developers test and run AI without specialized hardware.

There is a catch: this is a technical improvement that mostly benefits developers and advanced users. The average person won't see an immediate change unless they use apps built on this technology. Also, the update is specific to certain CPUs (x86 architecture), so Macs with Apple chips or phones won't benefit directly. Still, it's a step toward making AI faster and more accessible for everyone.