MHP Ensemble Hits 89.96% Accuracy on 6-Task E-Commerce Relevance

A new arXiv paper (accepted at SIGIR 2026) tackles a fundamental problem in e-commerce: how to build a single relevance model that handles six different entity pair relationship types—from query-product matching to product type similarity—each with varying data volumes and semantic requirements. Current industry practice relies on separate models per task, preventing knowledge transfer and producing inconsistent signals. The authors propose a unified multi-task framework using a shared 3-point relevance scale and compare three task routing architectures: text prefix routing, multi-head classification, and multi-head with private transformer layers (MHP). These are tested across both LoRA-adapted LLMs and fully fine-tuned cross-encoders.

The standout result is the MHP Ensemble, which achieves 89.96% accuracy on 453K test examples—the highest among all configurations. The study also reveals an important asymmetry: removing text prefixes without private layers causes severe performance degradation for decoder-only LLMs, while cross-encoders remain robust. This suggests that LLMs and cross-encoders encode task identity through fundamentally different mechanisms. Additionally, multi-task training yields up to 14% improvement on low-resource tasks compared to single-task baselines, demonstrating significant knowledge transfer benefits. The work provides practical guidance for building unified relevance models in large-scale e-commerce systems.