New Software Trick Makes Long AI Chats 23 Times Faster

Here's the problem. When you have a long back-and-forth with an AI — or let it grind through a multi-step task — it keeps "notes" about everything said so far. Those notes live in a small pool of super-fast memory on the graphics chip. Once the notes outgrow that fixed pool, the server either throws old notes away, redoes work it already did, or pauses your request entirely. That's why long AI sessions feel like they hit a wall.

The problem is worst for a popular kind of AI called a "mixture-of-experts" model. Think of it as a huge team of specialists where only two or three are working on any given word. The rest sit idle — but they still take up most of the fast memory. That's wasted space the conversation notes can't use.

A research team built a fix called VAMP. Instead of drawing one permanent line between "model weights" and "conversation notes," VAMP redraws that line while the system is running. When notes won't fit, it compares three options: shuffle idle model chunks into slower regular memory, discard cached notes and redo that work, or pause and reschedule the request. It picks whichever costs least, then uses a graphics-chip feature to redraw the memory map instantly — no copying data around.

They tested it on a large open model (Qwen3-Next-80B) running a 2,103-turn software-engineering task, replayed five times. The worst-case wait for a first response fell from 26.1 seconds to 1.10 seconds — 23.6 times faster. Total requests handled rose 20.7%. The catch: each individual word of output came out about 31% slower. So it's a big win for long, think-heavy jobs, and a slight loss for snappy conversation.