New Speed Trick Makes AI Chatbots Answer 82% Faster

When an AI chatbot writes a reply, it picks one word at a time. For each word, it checks its entire vocabulary — which can be hundreds of thousands of words — to see which one fits best. That check is like reading through a whole encyclopedia to find one fact: slow and memory-heavy. Researchers from the Institute for Advanced Research in Artificial Intelligence (paper authors Martin Loretz and Sepp Hochreiter) found a smarter way. They turned the word-picking step into a quick search, similar to using a librarian who points you to the right shelf instead of scanning every book. This "approximate retrieval" method finds a short list of likely words and then picks from that. For small AI models running on a regular CPU, this made the whole process up to 82% faster — with no noticeable drop in response quality.

Why does this matter to you? Speed like that means chatbots and AI assistants could feel almost instantaneous, even on a phone or a laptop. Right now, many AI services rely on huge, expensive server farms because generating text is so processing-heavy. This technique attacks the biggest bottleneck, which means smaller companies could run AI on cheaper hardware — and prices for AI services could eventually come down. It's especially good news for small models: AI that runs entirely on your device, without sending your conversation to a cloud server. That's better for privacy, too.

But there's a catch. The results were tested on CPUs with small models like Gemma 3 270M, not on the massive systems behind ChatGPT or Gemini. In rare cases, the search might skip a perfect word and pick the second-best one, though tests show the overall quality stays level. It's promising research — not yet a feature you'll see in your apps tomorrow. Still, it points toward a future where cutting-edge AI is faster, cheaper, and lives on your own devices.