ShareChat's post-stratification method slashes A/B test traffic by 45%

A new paper from ShareChat researchers tackles a major pain point in online experimentation: heavy-tailed monetization metrics like revenue or creator earnings. These metrics are dominated by a tiny fraction of users, making A/B tests noisy and statistically unreliable unless run on massive traffic. The team proposes a practical variance reduction framework that combines post-stratification with CUPED (Controlled-experiment Using Pre-Experiment Data). By leveraging pre-experiment covariates to adjust post-hoc, the method improves the sensitivity of ranking experiments without requiring additional traffic.

Deployed across ShareChat's ranking-driven monetization experiments, the technique achieved equivalent statistical confidence with roughly 45% less traffic than standard metrics. This translates to faster decision-making, lower risk from long-running tests, and more reliable conclusions for revenue-critical systems. The paper, accepted as an Industry Track paper at the 2026 ACM SIGIR Conference, also provides practical guidance on when post-stratification is appropriate, including design choices, guardrails, and limitations. For any team running online experiments with skewed monetization data, this offers a concrete, production-validated path to better power without extra traffic costs.