Google's Invisible Traffic Cop Makes AI Answers 17% Faster

Google researchers have published the details of a behind-the-scenes system called DLB, short for Distributed Load Balancer. Every time you ask an AI chatbot a question, some computer in a data center has to answer it. DLB is the software that picks which computer. It has been running inside Google for 22 months, handling millions of requests per second across thousands of different AI models.

The problem it solves is simple to describe. AI answers take unpredictable amounts of time — a one-line reply is quick, a long essay is slow — and the chips that do the work are scarce and expensive. Older systems relied on crude rules, like sending each new request to the next machine in line, and companies compensated by buying far more hardware than they needed. DLB instead lets servers trade information peer-to-peer about how busy they really are, and it learns to predict how long a request will take before sending it somewhere.

The results, measured during real migrations inside Google, were a 17% drop in the typical wait time and a 13% improvement in the slowest 5% of requests. In everyday terms, a two-second reply becomes roughly 1.7 seconds, and the occasional painful pause gets noticeably shorter. Because the same chips now handle more work, Google also needs less hardware to deliver the same experience.

The catch: this is Google's internal plumbing, not a product you can buy, so the benefit reaches you only through Google's own AI products. The gains are measured against Google's older system, not a neutral standard, and faster, smoother AI may simply encourage more usage rather than lower bills. Still, when AI runs your customer support, your search results and your email drafts, shaving seconds off every reply adds up across billions of daily interactions.