New Trick Makes AI Chatbots Answer Up to 50% Faster on Same Hardware

When you type a question into an AI chatbot, the model doesn't write the whole answer at once. It produces one word at a time, and every word requires a full trip through a very large, very expensive model. That is why long answers feel slow and why AI companies spend enormous amounts on chips. A popular fix is called "speculative decoding": a small, fast helper model guesses the next few words, and the big model checks the guesses in a single pass. When the guesses are right, you get several words for the price of one.

The problem is that this trick wastes effort when lots of people use the AI at the same time. Guesses that don't fit get thrown away, and the system pads out the work to keep things neatly aligned — like a restaurant kitchen plating empty dishes just so every tray looks the same. DScale, from a team publishing on arXiv, fixes this by adding a tiny extra component (about 112,000 adjustable numbers, minuscule next to an 8-billion-parameter model) that decides on the fly how many guesses are worth checking, and packs them in more tightly.

The results are meaningful for anyone paying for AI. Across four test datasets and 8 to 32 simultaneous users, DScale delivered about 44-49% more throughput than one leading method (DFlash), 22-38% more than another (DSpark), and 24-32% more than a third (Domino), while also cutting how long each request waits. Time spent on a single decoding step dropped by 31-52% on a common math benchmark. In plain terms: same computers, meaningfully more answers per hour.

The honest catch: this is a research paper, not a product. It was tested on Qwen3-8B and Qwen3-4B models running on Nvidia A100 chips, and it needs the small predictor to be trained for each model it accelerates. So expect this to show up first in the behind-the-scenes infrastructure of AI companies — as cheaper, snappier chatbots — rather than as something you can download and use today.