Nvidia's New AI Trick Makes Chatbots Twice as Fast for Free

Nvidia researchers have introduced a new way to make AI language models generate text much faster. The method, called 'probabilistic sampling for simple draft and MTP,' uses a two-step process: a smaller, faster model drafts a chunk of text, and the larger model verifies it. This is similar to how a junior writer might draft a document and a senior editor quickly checks it. The result is that the AI can produce responses in less time without sacrificing quality.

For everyday users, this means AI chatbots and assistants could become snappier. Imagine asking a question and getting an answer almost instantly, rather than waiting several seconds. This speed boost could make AI tools more practical for tasks like writing emails, translating languages, or even coding help. It also reduces the computing power needed, which could lower costs for companies and eventually for consumers.

The key innovation is that the drafting step now uses probability, not just picking the most likely word. This makes the drafts more varied and accurate, so the verifier accepts more of them. The team also fixed issues with random number generation to ensure the process is reliable. They added a fallback for grammar-constrained requests, so it works even when the AI must follow strict rules, like in legal or technical writing.

While this is a technical improvement, it's part of a broader trend: making AI faster and cheaper. As these optimizations roll out, you can expect AI assistants to feel more responsive and be available in more apps. The downside? It's still in the research phase, so it may take time before you see it in your favorite AI tools.