Their A/B test winner performed worse after rollout. I investigated why.
An independent article nobody knew about brought a surge of downloads during the test. I separated that cohort and found that the control performed better for the remaining organic traffic.
Why did the winner perform worse after rollout?
The A/B test showed a clear winner, and the team rolled it out. After rollout, the variant performed much worse than it had during the test. I investigated the difference.
An outside article overlapped with the experiment.
An independent article about the product, which nobody on the team knew about, had brought a large number of new downloads while the test was running. The original analysis had not checked total user numbers or the baseline behavior before the test.
Separate the article cohort from the other users.
I investigated the audience behind the result. When the users who came from the article were excluded, the control variant performed better for the remaining organic traffic.
The headline winner hid a different result for organic users.
The investigation explained why the original result did not carry over after rollout. The temporary influx of users from the article had changed the audience behind the test.
The control performed better for organic traffic outside that cohort. That distinction mattered when interpreting which variant had actually worked for whom.
Have a related problem?
Independent experiment review →Bring me
the problem.
An app that needs building. A migration that needs someone to own it. Numbers that don’t make sense. A product decision your team can’t resolve. Tell me what’s happening. I’ll tell you whether I’m the right person, how I’d approach it, and what the work would cost.
Email Alex ↗alex@x91tech.com