Single Experiment Review for a test you need to trust.
An independent review of one planned, running, or completed experiment. I check whether the audience, assignment, exposure, metrics, tracking, duration, and interpretation support the decision you want to make.
Send the setup or result as it is. I’ll reply with what is needed to scope the review.
A technically valid split can still produce the wrong decision when the tested audience does not match the audience you plan to roll out to.
Who this is for, and what changes after.
Product, growth, data, or engineering teams with one specific experiment that is being designed, is already running, or has produced a result that feels less certain than the dashboard suggests.
You get a clear confidence read, the issues that could distort the result, the corrections needed, and a recommendation on what decision the evidence can safely support.
This reviews one bounded experiment. It does not audit the team's wider experimentation system or search the full product journey for the right problem to test.
A 50/50 split is only one small part of a reliable test.
The useful population must enter the test, assignment and exposure must remain consistent, metrics must represent the outcome, and the result must apply to the audience the team intends to reach. A clean dashboard cannot repair a mismatch in any of those layers.
- What this prevents
- fixing symptoms instead of leaks
- metrics drifting away from user value
- experiments being interpreted without context
- roadmap work based on partial evidence
When a winning result needs a closer look.
A result that changes after rollout can have several explanations. I check the audience and implementation behind the comparison before advising what to do next.
The winner performed worse after rollout
An outside article brought a surge of downloads during the test. I separated that cohort and found that the control performed better for the remaining organic traffic.
Explore →Five experiments were affecting one another
I compared cohorts, group splits, and behavior in the raw data to identify which experiments changed behavior and how they influenced one another.
Explore →See the format of an experiment review
The A/B test sample shows the kind of findings and decision guidance you receive. It uses a fictional company and synthetic data.
What gets reviewed.
Decision & Hypothesis
Clarify the decision the test should support and whether the hypothesis connects to a meaningful product or business outcome.
Eligible Audience
Check who should enter the test, who should be excluded, and whether the tested population matches the intended rollout audience.
Assignment & Exposure
Review randomization, unit of assignment, repeat exposure, contamination risk, and whether users encounter the variants as intended.
Success & Guardrail Metrics
Check whether the primary metric represents the intended outcome and whether important side effects are protected.
Tracking Implementation
Verify that exposure, conversion, and supporting events represent the same behavior across variants and platforms.
Duration & Sample
Review traffic assumptions, test duration, sample sufficiency, and risks from novelty, seasonality, or unusual acquisition mix.
Segment Context
Identify segments that should be balanced or interpreted separately without turning the analysis into post-hoc fishing.
Result & Rollout Relevance
Assess what the result can support, what remains uncertain, and whether the conclusion applies to the planned rollout.
How the review works.
Share the test
Send the hypothesis, variants, audience rules, metrics, screenshots, implementation notes, and available results.
Inspect the setup
I trace the logic from eligibility and assignment through exposure, measurement, analysis, and rollout.
Separate evidence from risk
I identify what is reliable, what could bias the result, and what information is still missing.
Make the decision clear
You receive the corrections and the smallest responsible next move.
What I deliver.
Experiment Confidence Read
A plain-language assessment of what can be trusted, what needs caution, and what cannot yet be concluded.
Setup & Measurement Findings
Specific issues across audience, assignment, exposure, metrics, tracking, duration, and analysis.
Correction Plan
Concrete changes needed before launch, during the running test, or before acting on the result.
Decision Recommendation
A clear next move: run, adjust, extend, rerun, segment carefully, roll out, or avoid making the decision yet.
Have one experiment you need to trust?
Send the setup, result, or messy context. I’ll reply with what is needed to scope a focused review.