Researchers Made AI Chips Share Memory — Your Chatbot Could Get Faster

Every AI chatbot, image generator, and voice assistant runs on graphics chips called GPUs (the number-crunching engines of AI). Each chip has its own fixed pool of memory — think of it as desk space. Right now, if one chip's desk is overflowing while three others sit empty, the busy chip simply stalls. Nothing can be moved over. That's wasted money for whoever pays the cloud bill, and slower answers for you.

The new system, EMA, turns those separate desks into one shared table. Chips can borrow memory from neighbors and give it back the moment it's needed. The clever part is that borrowing doesn't feel slower: the software fetches data ahead of time (called prefetching), so a chip using someone else's memory performs just like it's using its own. And the chip that lends memory can always take it back instantly, so lending never hurts its own speed.

The results are notable. Individual users got up to 52% more work done per second. The shared system reached 96% of the performance of a machine with twice as many chips — meaning you get nearly double-the-hardware results without buying double the hardware. Waiting times stayed the same as if nothing had been shared. One author, Ion Stoica, co-founded the data company Databricks.

The catch: this is a research paper, not a product you can use today. It only works among chips inside the same physical server with very fast internal connections — it can't stretch memory across distant data centers. And sharing memory solves space problems, not raw computing speed. Still, the direction matters: squeezing more out of chips we already own is exactly what makes AI cheaper to run.