New Fix Stops One Busy Customer From Slowing Everyone's AI

Almost every AI chatbot you use — ChatGPT, Gemini, Claude, your bank's support bot — runs on shared computers in a data centre. Many different companies rent space on the same machines. That's efficient, but it creates a familiar problem: when one customer sends a flood of requests, everyone else's AI gets slower. Your answers arrive in choppy bursts instead of flowing smoothly. The paper calls this a "token latency" problem. A token is a small chunk of text — roughly three-quarters of a word — and each one takes a moment to appear on your screen.

Today's fixes try to be fair over the long run, like a supermarket giving each shopper an equal number of items per minute. But that doesn't stop your individual checkout from stalling. The new system, called FairInference, works more like a traffic light with per-car timing: it gives every token a deadline and enforces it. The authors describe this as a "δ-token fairness guarantee." In plain terms: if a word would take one second to generate when the AI is alone, it will take at most one second plus a small, fixed amount (that δ) even when the service is crowded.

Getting there was tricky because graphics chips — the hardware that powers AI — weren't designed to be sliced up finely among users. The team also had to account for the shared "KV cache," the AI's short-term memory of your conversation, which lives in the same crowded chip memory. Their scheduler handles both. In tests, it smoothed out the worst slowdowns and also handled more total requests than today's best systems.

The catch: this is a research paper, not a product. No major AI provider has announced it yet, and the guarantee applies to "well-behaved" clients — those that follow the rules. Still, it points to a future where the AI you pay for behaves the same at 9 a.m. on a Monday as it does at 3 a.m. on a Sunday.