Study: Two-sided A/B test isolation carries hidden engagement cost that persists at scale

Running A/B tests on two-sided marketplaces is tricky: changes to creators affect viewers and vice versa. To avoid cross-arm interference, many platforms use symmetric two-sided isolation, assigning matched fractions of creators and viewers to separate submarkets. But this thins each viewer's candidate catalog, and intuition says the damage shrinks as the platform grows. In 'The Price of Isolation' (arXiv:2608.04432), Yuanyuan Shen and colleagues show this intuition fails under heavy-tailed match quality. Using extreme-value theory, they derive tail-class loss laws: for light or bounded tails, engagement loss vanishes as candidate pools grow; under heavy tails, it converges to a size-independent constant—so even scaling the catalog by orders of magnitude doesn't eliminate the cost.

Empirical evidence comes from two production experiments on a platform with millions of active creators. A pure A/A traffic sweep revealed a measurable, depth-graded engagement cost; a one-sided catalog ablation showed per-viewer thinning drives the loss; and a tail index calibrated on a small exploration pool predicted the full-catalog effect accurately. The authors propose a preflight procedure that estimates isolation cost before launch, sizes traffic accordingly, and recommends a fallback design if predicted cost exceeds tolerance. For practitioners, the takeaway: isolation is a budgetable line item, not a free lunch. Paper: arXiv:2608.04432.