The Usual Trick for Shrinking AI Search Models Doesn't Work, Study Finds

Every time you search a photo app, get a product recommendation, or ask an AI assistant to find something, a special kind of AI called an "embedder" is doing the work. Its job is to turn words and images into numbers so computers can tell what's similar to what. These models are getting squeezed down — a process called quantization, which is basically saving a high-resolution photo as a low-quality JPEG so it fits on a phone instead of a data center. The paper asks a simple question: does the usual advice for squeezing them actually hold up?

It doesn't. The author tested five models from four different design families, across a range of squeeze levels, and found the standard playbook breaks down. The biggest surprise: the embedding table — the part everyone assumes you must protect, and the largest chunk of the file in several models — never turns out to be the thing you most need to save. At moderate squeezing, the differences between parts of the model are too small to matter. At extreme squeezing, the damage stops adding up predictably and depends entirely on which model family you're using. Two models squeezed equally hard can end up with wildly different results: one keeps about 66 percent of its accuracy, another drops to 1.3 percent.

There is good news buried in the results. A small model with 109 million parameters, squeezed hard, scored 78 on a standard accuracy test while taking up just 68 megabytes. Its much bigger teacher model — 0.6 billion parameters, 298 megabytes — scored only 64.5. So the little one won on both size and quality. That's the promise of shrink-to-fit AI: better answers in a fraction of the space.

The catch is real, though. That small model only won at the specific task it was trained for. Take it somewhere new and the advantage may evaporate. The takeaway for anyone building AI products: don't assume the old squeezing rules transfer from big chatbots to search and recommendation models. Test your own. The paper also notes all its file sizes are real measured bytes, not estimates — a small but welcome bit of honesty.