Researchers' New Trick Lets Cheap AI Chips Handle Huge Documents

Many AI tools work by looking things up before they answer — a technique called RAG (letting AI search your documents first). To keep costs down, companies often shorten, or "compress," those documents before feeding them to the AI. But the researchers found a catch they call the Compression Paradox: squishing the text can actually slow everything down or exhaust the chip's memory, causing crashes.

Their fix is a traffic cop for text. Before each request, a simple software "router" checks three easy-to-measure signals — how long the text is, how dense the wording is, and how varied the vocabulary is — plus how much memory is left on the chip. It then sends the request down one of three paths: use the text as-is, compress it with a small AI model, or just trim it with old-fashioned keyword filtering.

The key discovery was a tipping point: on NVIDIA's T4 chip (a older, low-cost model with 16 GB of memory), documents longer than roughly 4,330 words are better handled by keyword trimming rather than AI compression. Across tests on unseen documents, the router had zero memory failures and matched the ideal choice 88.5% of the time. It scored 5.2 points higher than always using keyword trimming — with no extra memory or training required.

So what? Better routing means capable AI can run on hardware that costs a fraction of cutting-edge chips. That could translate into cheaper per-query pricing for AI document search, more AI features running privately on your own laptop or company server instead of the cloud, and fewer frustrating "something went wrong" errors when you upload a 60-page PDF. The catch: the exact tipping point was tuned for one specific chip, so other setups need their own calibration.