Nvidia's New AI Trick Makes Chatbots Faster and Cheaper to Run

If you've used a chatbot like ChatGPT, you've interacted with a large language model. Many of these models use a technique called 'mixture of experts' (MoE), where the model is divided into many specialized parts, and only a few are used for each query. This saves energy but can cause bottlenecks if some experts are used more than others. Nvidia's new ISER-MoE method tackles this by copying the most popular experts to multiple chips, so no single chip gets overwhelmed.

The key innovation is using the high-speed NVLink switch to broadcast these copies efficiently. Instead of sending data multiple times from one chip, it sends once and the switch distributes it. This reduces delays and speeds up processing. In tests, the method removed 98-99% of the excess load and improved throughput by 17.4% on a math benchmark and 7.2% on agentic prompts.

For everyday users, this means AI services could respond faster and handle more requests simultaneously. For companies, it could lower the cost of running AI, potentially making services cheaper or more accessible. As AI becomes more integrated into daily life, such improvements help ensure smooth and efficient experiences.

While this is a technical advancement, it's part of a broader trend of optimizing AI infrastructure to meet growing demand. The catch is that it requires specific hardware and may not be universally applicable, but it's a step toward more efficient AI.