RaaS paper reveals LLM integration trade-offs in industrial retrieval pipelines

A new paper from Liu et al., titled "Retrieval-as-a-Service: A System-Oriented Analysis of Industrial Retrieval Pipelines in Web Systems," shifts focus from algorithmic advances to the infrastructure realities of production retrieval systems. The authors introduce a unified RaaS pipeline abstraction that models retrieval as a multi-stage service: high-efficiency candidate generation (e.g., inverted indices, lightweight ANN), embedding-based semantic matching (using dense vectors or learned representations), and resource-aware re-ranking (balancing accuracy against latency budgets). They emphasize how real-world constraints—sub-100ms latency targets, petabyte-scale index sizes, and heterogeneous hardware—force trade-offs that pure algorithm research often ignores.

The paper further examines the integration of LLM-based retrieval mechanisms, such as using large models for query rewriting, dense embedding generation, or direct re-ranking. While LLMs improve semantic relevance and handle zero-shot queries, they introduce significant computational overhead and latency spikes (often 10–100x slower than traditional lexical methods). To mitigate this, the authors discuss caching strategies, model distillation, and tiered retrieval where LLMs only process a small candidate pool. The survey concludes with practical guidelines for designing scalable, efficient, and QoS-aware RaaS architectures, making it a must-read for engineers building or operating retrieval systems at scale.