AutoRelAnnotator's model cascade halves search annotation cost without accuracy loss

Relevance annotations are critical for training search ranking models, evaluating NDCG, and performing root cause analysis. However, human annotation is slow and expensive, while off-the-shelf LLMs often struggle with domain-specific tasks. To address this, researchers introduce AutoRelAnnotator, a calibrated model cascade that systematically routes queries through progressively larger fine-tuned classifiers. The key insight is that accuracy and cost can be optimized orthogonally: domain-specific fine-tuning drives accuracy, cascading controls cost, and per-class isotonic calibration adds a small but reliable gain.

The paper decomposes the gains: fine-tuning contributes 20 accuracy points, while the cascade is accuracy-neutral but halves compute cost. Per-class isotonic calibration adds a statistically significant +0.6 points over the strongest baseline. The system was validated in production across six offline use cases, processing over 150 million annotations. This enables faster experimentation cycles for search and advertising systems, offering a scalable path to high-quality offline annotation without the latency and expense of human labelers.