Free AI Tool llama.cpp Just Got Twice as Fast at Long Documents

llama.cpp is a free piece of software, downloaded about 129,000 times over on GitHub, that acts as the engine for running AI models directly on your own machine — no subscription, no sending your data to a company's servers. Its newest build, numbered b11140, arrived on 23 September and is mostly a set of under-the-hood speed fixes. This matters because 'local AI' is the main alternative to paying $20 a month for a cloud chatbot.

The specific fix concerns something called 'sparse attention,' which is the AI equivalent of skimming. When you hand an AI a 200-page document, it can't afford to carefully reread every word, so it builds an index of which chunks matter and only looks at those. This update makes building and updating that index faster on Nvidia graphics cards. The developers report that one small step in that process fell from 46 to 17 microseconds, and a larger batched step at 49,000 words of context fell from 586 to 244 microseconds.

In practical terms, microseconds sound trivial, but they multiply. Long-document work involves thousands of these tiny operations, so shaving each one speeds up how fast you get an answer when you paste in a contract, a research paper, or a lengthy chat history. Anyone using a local AI app — and many popular desktop apps are built on top of llama.cpp — benefits without doing anything.

The catch is real and worth stating. This is a pre-release build aimed squarely at Nvidia GPU owners; if you're on a Mac, an AMD card, or a plain CPU, you won't feel it. The commit also references a model called 'dsv4,' which hasn't been publicly released, so it's partly groundwork for something coming later. And because the numbers are measured on specific test shapes, your real-world speedup could be smaller — or invisible — depending on your hardware.