Microsoft's HARNESS-LM slims down search AI by 27x latency with 98% accuracy

Microsoft researchers (Bing Ads) have developed HARNESS-LM (HLM), a three-phase training recipe that transfers the capabilities of large-scale sponsored search retrievers into compact, cost-efficient models. The approach starts with fine-tuning a billion-parameter SLM (e.g., Qwen3-Embedding-8B) as a high-performance teacher. Phase two aligns query representations via an L2 distillation objective to a student encoder with under 600M parameters. Phase three applies contrastive refinement to optimize the student for retrieval performance. The team also conducted a comprehensive empirical study of key design choices like alignment objectives, embedding dimensionality, model scale, and architecture.

On a real-world Bing Ads evaluation benchmark, HLM recovers over 98% of the reference retriever's precision while delivering up to 27x lower online query-encoder latency and 20x higher throughput on NVIDIA A100 GPUs. An online A/B test on Bing Ads further validated the approach: the deployed 190M parameter model produced a +1% Revenue, +0.6% Impression, and +0.4% Click uplift over the existing ensemble of retrievers. This demonstrates that even relatively lightweight models can match or exceed the performance of much larger systems in production, making them ideal for high-throughput, latency-sensitive sponsored search environments.