Pre-inference routing cuts extraction costs 77% with minimal quality loss

Most document extraction systems use a single model for every input, which is wasteful for easy documents and inadequate for hard ones. Sreerekha Rajendran's arXiv paper introduces a pre-inference routing approach that predicts each document's difficulty from cheap, observable signals like image quality and layout, then selects between a budget model and a larger, stronger extractor. The method works only when two conditions hold: the cheap model fails often enough to matter, and those failures are predictable from visual features. When both conditions are met, the calibrated router reduces cost by 31-33% on receipts and 77% on degraded ad-buy forms, while staying within 0.02 F1 of always using the large model. A small labeled pilot can accurately forecast whether routing will succeed, and the results hold across two model pairs with 5x and 3x cost differences.

The study tests five document genres, revealing that routing does not help when documents are already easy, such as clean digital invoices or nutrition labels. Strikingly, a simple bag-of-words router performs as well as engineered features, suggesting the main bottleneck is the genre characteristics rather than router complexity. The router must be retrained for each dataset and does not transfer across datasets, even within the same genre, limiting its use as a plug-and-play component. However, for high-volume pipelines with heterogeneous document quality, the cost savings are substantial: up to 77% on degraded forms with negligible quality trade-off. Rajendran's work provides a practical framework for deciding when routing is worthwhile, turning an intuitive idea into an evidence-based engineering practice.