Five A/B tests were running. I untangled how they affected each other.
I compared cohorts, test-group splits, and behavior in the raw data to identify which experiments changed behavior and how they influenced one another.
Which test was changing what?
The company had five different A/B tests and did not know how they affected one another. Looking at their results separately was not enough to explain the behavior.
Compare the groups in the raw data.
I investigated the raw data, compared cohorts and the splits between experiment groups, and examined the behavior of each group. I worked through which tests changed behavior and how those changes affected other tests.
One test distracted low-intent users from another.
One experiment changed the audience reaching another experiment by distracting low-intent users. The users who actually used the other variant continued to use it.
The apparent interaction needed to be understood through the users reaching the test and their intent, rather than the aggregate result alone.
Explain the behavior behind the interaction.
The analysis identified how one test changed the audience of another. It distinguished a change in who reached the experience from a change in behavior among the people who used it.
Have a related problem?
Independent experiment review →Bring me
the problem.
An app that needs building. A migration that needs someone to own it. Numbers that don’t make sense. A product decision your team can’t resolve. Tell me what’s happening. I’ll tell you whether I’m the right person, how I’d approach it, and what the work would cost.
Email Alex ↗alex@x91tech.com