New AI Trick Makes Chatbots Reply 20% Faster on the Same Chips

When an AI chatbot writes a sentence, it isn't really 'thinking' hard. It's mostly fetching data. Every single word it produces requires reading the model's entire memory once. That fetching — not the math — is the slow part. Picture a chef who cooks instantly but must fetch every ingredient from a warehouse across town for each dish. The kitchen sits idle, waiting.

The fix is something called a 'megakernel' — one giant, tightly packed set of instructions that keeps the processor busy instead of pausing constantly. The problem: these are brutally hard to write by hand, and automating them usually breaks correctness. The researchers' answer, ForgeMegakernel, hands the job to AI coding agents. They follow a checklist of ten stages and an independent verifier that checks the work in progress, like a proofreader catching errors before a book goes to print. The result is custom-tailored instructions for each specific model.

They tested it on 14 decoding tasks across eight model families, from tiny 0.6-billion-parameter models to 13-billion-parameter ones. The generated code ran 21% faster than SGLang, a widely used tool for serving AI, and 54% faster than an existing megakernel compiler. It used 50% to 86% of the available memory bandwidth. Every one of the 14 tests beat the standard setup while giving equally accurate answers.

So what does this mean for you? Faster answers, mostly. When AI companies spend less computing power per reply, those savings tend to trickle down as lower prices, more generous free tiers, and less energy burned per question. The honest catch: these are 20-to-50% improvements, not miracles, and it's a research paper, not a product you can buy today. It also speeds up only the 'answering' step, not the far more expensive training. Still, it's a striking signal: AI is now good enough to optimize its own plumbing.