New Research: When to Upgrade Your A/B Tests to Smart AI Testing
Your company's A/B tests may be leaving money on the table — here's when.
Almost every company runs A/B tests: show half your customers version A, half version B, and see which one sells more. It's simple and reliable, but slow and rigid. A newer approach called "adaptive experimentation" — powered by contextual bandits (software that shifts traffic toward whatever is working, and tailors choices to each customer) — can change the test while it's running. The catch is that it's more complex, harder to trust, and can go badly wrong. So when is it actually worth switching?
A team of researchers from Brazil, publishing at the BRACIS 2026 conference, offers a practical answer. Their trick is to squeeze the answer out of data you already have. Using historical A/B test results, they estimate how different types of customers responded, then simulate how a range of adaptive and non-adaptive strategies would have performed. This is called off-policy evaluation — judging a strategy you never actually ran, using data you already collected. In plain terms: it's a test drive for a test you haven't launched yet.
Their conclusion is refreshingly clear. If customers genuinely differ — some love the discount, others ignore it — then adaptive, personalized testing beats the old fixed split. If everyone responds roughly the same way, the fancy approach adds complexity for almost no gain. They confirmed this on synthetic trials where they knew the true answers, plus three well-known public datasets (Hillstrom, Criteo Uplift, and LaLonde) used widely in marketing and economics research.
Why does this matter outside academia? Because companies burn real money and real weeks on experiments. This method gives teams a cheap, low-risk way to decide up front: stick with the simple test, or invest in the adaptive one — and if adaptive, which version. It won't tell you what your customers want. But it can stop you from paying for sophistication you don't need.
- A/B testing (showing two versions to two groups) is being challenged by adaptive testing that adjusts while it runs — but only pays off when customers respond differently to different things.
- The researchers' method uses data companies already have from past tests, so no new experiment is needed to make the decision.
- It worked across three standard public datasets used in marketing and economics, suggesting real-world businesses can apply it.
Why It Matters
Businesses can decide cheaply whether smarter testing is worth the cost — saving money on experiments that won't pay off.