New Software Makes AI Chatbots 31% Faster on the Same Chips

AI chatbots run on specialized chips that come with a small pool of extremely fast memory, called HBM (short for high-bandwidth memory). That pool is tiny and expensive, so when a model is too big to fit, today's serving systems copy data from the computer's regular memory into the fast pool before using it — like hauling books onto your desk before you can read them. All that copying burns the fast memory's speed and leaves the regular memory largely unused.

BOOST takes a different approach: it uses both pools at the same time, in proportion to how fast each one is. The trick is timing — it watches how the chip chops work into batches, then places data so each batch pulls from both fast and slow memory simultaneously. The researchers plugged it into vLLM, a popular free tool many AI companies already use to serve models, and tested it on an Nvidia Grace Hopper system. Importantly, no changes to the AI model itself were needed.

The results: in high-traffic serving, BOOST pushed throughput up by 31% on average, beating the old copy-first method by 15 percentage points. At a fixed batch size, the time to produce each word of output improved by 4.3%, while the old approach actually made things 6% slower. In plain terms, a server running BOOST can field more users at once, and each reply comes back a little quicker — using hardware you already have.

The catch is that this is a research paper, not a shipping product. It was tested on one specific kind of Nvidia hardware, and the per-reply speed gain is modest — the bigger win is handling more people at once. Still, it points to a pattern worth watching: much of the next wave of AI progress may come from using existing chips more cleverly, rather than buying new ones.