New Tool Lets Supercomputers Run AI for Thousands of Users at Once
Faster, cheaper AI for everyone — and less expensive hardware sitting idle.
Supercomputers are the most powerful computers on Earth, but they're built for science — simulating climates, folding proteins — not for answering questions from ordinary people. Serving AI on them usually takes weeks of expert tinkering: special job schedulers, custom launch scripts, and patches for each machine. A research team from Argonne National Laboratory and the University of Chicago fixed that with ExaServe, a free add-on you install with a single command. You describe what you want in a short settings file (a YAML file — just a plain list of preferences), and ExaServe does the rest.
On the Aurora supercomputer, they tested it from 1 machine up to 256 machines running 3,072 copies of an AI model at once. Plain question-and-answer requests scaled almost perfectly: 27,100 requests per second, or about 3.8 million word-chunks per second. That's roughly 30 novels of text generated every second. For comparison, a busy commercial AI chatbot handles a few thousand requests per second on far more expensive, purpose-built hardware. The result hints that existing supercomputers could absorb a lot of AI traffic without anyone buying new data centers.
The catch appeared when the AI streamed its answer word by word — the way ChatGPT types at you. A single central router that manages all the streaming connections became a traffic jam, capping throughput at about 4,700 requests per second even though the AI machines themselves were barely sweating. The team also found that starting up 256 machines takes around 30 minutes because the coordination software slows down sharply as it grows — a problem that gets worse, not better, the bigger you go.
Why does this matter to you? Streaming is how nearly every consumer AI product feels fast and responsive. If the world's biggest computers can't stream efficiently, that's a real ceiling on how cheaply AI can be delivered at massive scale. The paper is a research demo on one specific machine, so don't expect your chatbot to speed up next week — but it shows both a promising path and a specific bottleneck worth fixing.
- ExaServe is free software that installs with one command and turns a supercomputer into an AI answering service, replacing weeks of expert setup.
- In tests on the Aurora supercomputer, it handled 27,100 questions per second across 256 machines — but only about 4,700 per second when answers were streamed word by word.
- Streaming is the slow spot because one central router gets overwhelmed, and starting up the biggest setups takes roughly 30 minutes.
Why It Matters
Cheaper, faster AI at scale could mean lower prices and less need for new data centers near you.