Two-Sided Prioritized Ranking: A Coherency-Preserving Design for Marketplace Experiments

A team of researchers has published a new paper titled 'Two-Sided Prioritized Ranking: A Coherency-Preserving Design for Marketplace Experiments' on arXiv, proposing a novel solution to a persistent problem in online marketplace experimentation. The core issue is 'interference' in A/B tests: when platforms like Amazon, Uber, or Expedia test price changes, altering one item's price inevitably affects demand for other items in a ranked list, biasing results. Traditional user-level A/B tests also fail because they violate 'platform coherency'—the requirement that all users see consistent prices and availability for the same item.

The researchers' solution, Two-Sided Prioritized Ranking (TSPR), cleverly exploits 'position bias'—the well-documented tendency for users to pay disproportionate attention to top-ranked items. Instead of showing different prices, TSPR randomizes both users and items into groups and then reorders search result lists. For one group of users, items with the new 'treatment' price are prioritized higher in the ranking. For the other group, items with the old 'control' price are prioritized. All users see the same items at the same prices, but their exposure to the treatment varies based on where items appear in their personalized list.

This design preserves platform coherency while creating the variation needed to measure the 'total average treatment effect' of a price change. The team validated TSPR using semi-synthetic simulations based on real Expedia hotel search data. Their results showed that TSPR significantly reduced estimation bias compared to other coherency-preserving experimental designs and maintained strong statistical power, making it a practical and robust method for real-world deployment.