Taobao's PILOT agents beat human experts in e-commerce experiments
PILOT achieved +1.6% Core IPV while requiring zero human oversight in Taobao's A/B tests
Deep Dive
On Taobao's platform, PILOT (Proactive Insight Learner for Online Tree-Experiments) — an LLM-agent framework — was deployed across 5 experimental buckets. Compared against ROAM, a reactive optimization agent, PILOT achieved up to +1.60% Core IPV and +0.96% transaction count, and raised search efficiency from 53.3% to 93.3% (+40 pp), with no human intervention throughout the experimental cycle.
Key Points
- PILOT increased Core IPV by 1.60% and transaction count by 0.96% in Taobao's A/B tests across 5 buckets
- The autonomous framework runs without human intervention while enforcing safety and statistical boundaries
- Search efficiency improved from 53.3% to 93.3% (+40pp) compared to reactive agents
Why It Matters
This demonstrates AI systems can autonomously optimize billion-dollar e-commerce platforms at scale, replacing human-led experimentation with automated, data-driven decision making