Free AI Tool Runs Faster, Adds New Models and Features

llama.cpp is a popular free, open-source project that lets you run large language models (AI that writes text) on your own hardware—like your laptop or a home server—instead of paying for cloud services. Version 0.7.0 brings a bunch of improvements that make local AI faster and more capable. The biggest speed boost comes from a new GPU cache for 'MoE experts' (a way AI models split up their knowledge). When parts of the model are offloaded to your computer's main memory, this cache can make the AI generate text up to 2.2 times faster. That means quicker responses and less waiting.

The update also adds support for many new AI models. Some can handle multiple types of input: text, images, and audio. For example, EmbeddingGemma-2 can understand text, pictures, and sound. Others like Nemotron Super 3.5 VL are designed for vision and language tasks. There are also new coding models like maion-coder. This means you can run a wider variety of AI applications locally, from chatbots to image recognition, without needing an internet connection.

On the server side, the chat system got a major overhaul. The new chat session API makes it easier for developers to build AI chat interfaces. The web UI now includes a full models manager, so you can download and switch between models more easily. The default port changed to 9931, which might require a small adjustment if you use the server. Overall, these changes make llama.cpp more powerful and user-friendly, especially for those who want to run AI on their own terms.