New Trick Cuts AI Training Time by 70% Across Far-Flung Data Centers

Big AI models like ChatGPT are built by thousands of specialized computer chips working together, often spread across data centers in different states or countries. The problem: those chips run at different speeds, and the connections between far-apart buildings are slow. So the fast chips finish their share and then sit waiting for the slow ones. In this study, fast chips wasted up to 21% of their time waiting, and everyone spent about 66% of their time just coordinating instead of actually learning.

A team of researchers from Korea and China built a fix called GeoMesh. Instead of giving every chip the same amount of work, it measures each one and hands out tasks that match its speed — think of a group project where the fast writers take more pages instead of twiddling their thumbs. That keeps everybody busy at once, with no chip running ahead or falling behind.

The second trick is talking less. Chips normally send big streams of numbers to each other constantly. GeoMesh instead sends a tiny summary — just a direction and a rough size, about 32 times smaller — so the slow long-distance links stop being a bottleneck. Combined, these two ideas cut the time to reach a target quality by up to 70%, while keeping the model just as accurate.

Why should you care? Training AI is enormously expensive, and that cost shows up in subscriptions, API prices, and which products companies can afford to build. Making training faster and less wasteful could mean cheaper AI tools, and companies able to build them using ordinary hardware in ordinary locations instead of one giant, costly campus. The catch: this is a research paper tested in simulations using real cloud-network data, not a product you can use yet — and it only speeds up the training step, not the day-to-day running of AI apps.