New AI Method Lets Chatbots Speed-Read 500,000 Words at Once

When you paste a long contract, report, or book into an AI chatbot, the AI has to hold all of that text in its short-term memory while it writes each word of its answer. That memory gets huge as documents get longer, and re-reading it over and over is the main reason long-document AI is slow, expensive, and sometimes just plain forgetful. A new research paper introduces Elastic Threshold Attention, a method that lets the AI decide, moment by moment, which parts of a document are worth re-reading and which parts it can safely ignore.

Earlier attempts at this were crude. They either skipped text using fixed rules — which threw away details the AI later needed — or kept everything, which was slow. This new approach is different because the AI learns the skipping policy itself during training. When a question requires careful searching or step-by-step reasoning, it automatically widens its focus. When a task is routine, it trims. The researchers say their 1.45-billion-parameter model matches standard, full-attention models on language tasks, common-sense reasoning, and finding a needle in a haystack of text, while doing roughly 85% less work during training.

At use time, they wrote custom software for the chips that run AI and measured up to 2.5 times faster response generation on documents up to 512,000 words, plus another 27% cut in computing for specialized setups. That translates directly into money: long-document AI is billed by the amount of computing it consumes, so making it faster and leaner makes it cheaper to run.

The honest catch is that this is an academic paper, not a shipped product. It uses a small 1.45-billion-parameter model, and results need to be confirmed when scaled up and by independent teams. Still, it points clearly in one direction: AI that handles your entire book-length context quickly, affordably, and without quietly forgetting page 200.