New Trick Lets AI Read Whole Books 6x Faster, Same Answers

When you paste a long document into an AI chatbot — a 200-page contract, a stack of medical records, a whole book — the model doesn't start answering immediately. First it reads the entire thing, word by word, figuring out which parts relate to which. That reading step (called "prefill") is usually the slowest part of your wait, and it gets worse the longer your document is. Researchers at arXiv have now published a method that skips most of that reading while barely affecting the answers.

The trick is a smarter way to skim. Older shortcuts chop a document into chunks and judge each chunk by its average importance — but averaging is dangerous. A single crucial sentence, like "termination fee applies," can get buried among dozens of ordinary ones and be wrongly skipped. The researchers call this "mean dilution," and their fix adds a second opinion: alongside the average, a "rescue" check looks for any chunk that might hide one unusually important line, and keeps those in. Crucially, the method works on existing AI models without retraining them.

The numbers are striking. On a high-end Nvidia H100 chip, the reading step ran about 20 times faster in one test, nearly 12 times faster inside a popular serving system, and about 6 times faster in overall response time for a 128,000-word prompt. Quality barely moved: 88.65 versus 89.52 on a long-context accuracy test. For you, that means near-identical answers with much less waiting.

The catch is that this is still a research paper, tested on specific models and specific hardware. The speedups won't magically appear in your favorite chatbot tomorrow — companies have to adopt and engineer it. And a 6x faster reading step doesn't mean a 6x faster chatbot overall, since writing the answer still takes time. Still, the direction is clear: long documents are getting cheaper to process, which usually means faster, less expensive AI for the rest of us.